How to Choose an Evaluator

Finding the right evaluator can be crucial to the success of your project.  Take the time before submitting your proposal to select the right evaluator.
What to look for in an effective evaluator

 

Knowledge, skills and experience

  • Quantitative expertise and experience including using SPSS, Stata, or R to conduct descriptive and inferential analyses; collecting and analyzing social networks.
  • Qualitative expertise and experience including using qualitative software such as NVivo and Dedoose to conduct content analysis.
  • Uses a mixed methods approach to triangulate findings.
  • Conducts evaluations through all phases:
    • Front end planning (evaluation scope of work, logic model, SWOT analysis, timelines, benchmark/milestone development, instrument selection and development)
    • Formative/process evaluation (evaluating the usefulness of all project activities, progress made towards goal achievement, and potential for sustainability)
    • Summative/outcome evaluation (evaluating impacts and outcomes of the project on participants, the institution, and the broader community)
  • Data-visualization capabilities to present results in multiple formats including tables, graphs, and graphics.
  • Utilization-focused approach including building skills, knowledge, and abilities of staff and stakeholders and facilitating use of evaluation findings.

Characteristics

    • Collaborative
    • Leader
    • Flexible
    • Reliable
    • Responsive to changing needs
    • Credible
    • Ability to problem solve
    • Reflective

Resources

      • Has sufficient time to conduct a thorough evaluation
      • Has sufficient well-trained personnel, necessary equipment, and infrastructure to complete all aspects of the evaluation such as primary data collection (e.g. observations, interviews, online and paper surveys) and in-person meetings.
How to determine effectiveness

Look at past work – Ask for samples of a logic model, fact sheet or newsletter, PowerPoint Presentations, evaluation briefs and reports. The evaluator should be able to present information in a clear, concise, easy-to-understand, and jargon-free manner.

Check references – Ask for a list of all projects the evaluator has conducted and request permission to contact whomever you choose.

Ask about the budget – Cheaper is not better.

      • The budget must be sufficient to conduct a quality evaluation. Adequate evaluation budgets range from 5-20% of the funded grant amount.
      • Generally, the smaller the funded amount, the higher the evaluation percentage because whether a project serves 5 people or 500 people, the evaluator must spend the same amount of time developing evaluation instruments, collecting and analyzing the data, and developing the report.
      • There are several tiers of evaluation costs:
        • For a descriptive evaluation using a pre-posttest design, 5-8% of the program budget is reasonable
        • Add a control or comparison group and the cost rises to 9-15%
        • Use a randomized experimental design with longitudinal follow-up and the cost will be up to 20%.
      • Remember that consultants get paid for their time as well as their expertise—every call, meeting, and presentation uses valuable contract time. Facilitate their work by providing data, participant email addresses, and connecting them with the right internal personnel. If you want or need more than what is outlined in the agreed-upon scope of work, expect to pay more for it.

Ask questions about the evaluator’s experience and approach:

      1. Tell me about your experience in evaluating grant programs and types of projects you have evaluated.
      2. How long has your organization been in business?
      3. Are you familiar with the standards and guidelines of the American Evaluation Association’s Guiding Principles for Evaluators (eval.org/Publications/GuidingPrinciples.asp)?
      4. Will you provide us with a scope of work including a timeline of deliverables?
      5. Describe how you work with the grant program staff and constituents.
      6. What is your experience with projects activities like ours?
      7. We want to know X about our program. Have you evaluated this type of program before?
      8. What processes and evaluation approaches would you use for our project?
      9. We have a budget of XX. What type of evaluation plan would you recommend?
      10. What other staff and resources will you utilize in conducting your evaluation?
      11. Do you subcontract for any services? If so, can you provide resumes of subcontractors?
      12. How rapidly do you provide feedback, and in what form?
      13. Can you provide us with a variety of work samples?
      14. Can you provide us with a list of grants you have evaluated and project directors’ contact information?