NSG 6999 Discussion Evaluation Plan
NSG 6999 Discussion Evaluation Plan
This week you will be working on an evaluation plan that
will be attached to your EBP in Week 9.
Submit your Evaluation Plan to the DB –listed as 1-2-3…
As you work on your capstone project proposal, you will want
to share your progress with your peers and instructor and seek or provide
guidance or share insights. By due date assigned, go to the Discussion Area and
post responses to the discussion question. All responses should be posted to
the appropriate topic in this Discussion Area. It is important to support what
you say with relevant citations in the APA format from both the course
materials and outside resources. Include the South University online library in
your research activities utilizing not only the nursing resource database, but
also those pertaining to education, business, and human resources.
Supporting one—Challenging one.
By the end of the week , comment on the responses of at
least two other students by supporting a minimum of one post and challenging a
minimum of one post. You will want to
focus on their point of view, asking pertinent questions, adding to the
responses by including information from other sources, and respectfully
challenging a point of view, supported by references to other sources. Be
objective, clear, and concise. Always use constructive language. All comments
should be posted to the appropriate topic in this Discussion Area. Please only
start a new thread with your original post. Hit reply to respond to a pee
Elements of an Evaluation Plan
An evaluation plan should be an integral part of your overall written plan for a quality reporting project. To support the planning of an evaluation, this page covers the following topics:
- Purpose of the Evaluation
- Evaluation Questions
- Evaluation Criteria
- Timetable and Work Plan
- Collecting Data for an Evaluation
- Data Collection Methods To Answer Evaluation Questions
- Data Collection Tools and Activities
- Data Analysis
- Reporting Evaluation Findings
Purpose of the Evaluation
To clarify the purpose of your evaluation, start by identifying what you need to learn in the short and long term. Think specifically about the decisions you and your partners are facing and when they have to be made. Key issues include:
- What are you hoping to learn from the evaluation?
- What decisions do you expect to make as a result of the evaluation?
- When do you hope to make those decisions?
Since your resources are sure to be limited, answering these questions will help to set priorities for learning.
Evaluations are most useful when they inform key decisions by answering the right question at the right time. What specific questions do you need to answer to adequately inform your decisions? Note that you may have several questions and that different questions may be appropriate to ask at different stages of your effort. For example, you will probably need answers to questions about your process sooner than you need answers to questions about results.
The number of questions you can address depends largely on the time and resources available. It also depends on whether you can save money by using the same data collection methods to gather the answers to more than one question at a time. For example, you might use a single community survey to address questions about whether your audience was aware of the report, sought it out, or used it. But this type of survey probably wouldn’t work for determining whether people understood the report.
To properly evaluate your efforts, develop specific criteria for success. Here are some issues to consider:
- What would count as success in reaching your audience?
- What will you consider a successful process?
- What will you consider a successful result?
- How would you determine whether someone had used your information?
- How would you know whether you did enough outreach?
The credibility of your evaluation with various stakeholders will depend in part on whether you define success in a way that resonates with them. They may have different points of view about the most important criteria for success. Make sure you get their input and come up with a clear set of criteria that reflect a shared vision. You might find that clarifying your criteria leads to useful, if sometimes thorny, discussions about exactly what you are trying to achieve, for whom, in your initiative.
Timetable and Work Plan
- What are the priorities of the evaluation effort?
- When do you need to get started on different phases of the evaluation?
- When do you expect to complete each phase?
- Who’s responsible for meeting each deadline?
- Who will monitor the evaluation process to see whether midcourse corrections are needed?
Starting Early on Data Collection
Planning your evaluation as early as possible makes it easier to start on your assessment when you want to. People who start late often find themselves playing “catch up” and cannot actually get the information they need.
As early as possible, decide when you will start work on collecting feedback. If you are evaluating your processes, you need to move quickly to gather the data you need. If you are evaluating your results, you may also need to start early if you hope to collect data on the situation before your report is issued. This information is often called baseline data.
However, even if you are well along in your efforts, and have not been able to focus on evaluation yet, you can and should start as soon as possible. If you are in this for the long haul, you need to harness evaluation tools to help the project move forward in the right direction as you get more sophisticated and perhaps more ambitious.
Collecting Data for an Evaluation
How will you measure whether each of your criteria has been met? When you’re thinking about what data to track, keep in mind that the things that are easiest to count are not necessarily the most informative. For instance, the number of reports mailed out to enrollees doesn’t tell you whether they read it, understood it, or used it.
When you develop your plan, answer these questions as well:
- Are there existing or standard measures or will you have to devise your own?
- What data sources will you use?
- Are some data already available that you can use? Most of the time, there is relatively little existing information relevant to quality reports. This means you will need to collect “primary” data from a variety of sources. Some of your primary data will be qualitative in nature; some will be quantitative.
One important thing to consider is whether you are collecting data on individuals or groups/organizations:
- If you collect data on individuals, you will likely focus on their
- Attitudes, beliefs, and preferences.
- Experiences and responses.
- Behavioral intentions (what they intend to do in the future).
- Actual behaviors.
- When you collect data about groups or organizations, you may also collect data on their
- New initiatives.
When you collect data about groups or organizations, you are typically collecting the data from individual people in the group or organization who are knowledgeable about the group or organization in question. These people are sometimes called “key informants.”
Data Collection Methods To Answer Evaluation Questions
How will you collect data on your measures? You are likely to be using a mix of qualitative and quantitative methods in your evaluation as well as perhaps tapping into existing data, especially if you are evaluating a Web-based report.
This page provides brief descriptions of several useful data collection methods for evaluating public reports. The method you use depends on the question you are asking as well as the time, resources, and talent that you have available. You must also consider what will be credible to the audience for your evaluation findings.
Matching Evaluation Questions to Data Collection Methods
A key decision in any evaluation is what data collection method to use to answer your evaluation questions. Here are some examples of how to fit a data collection method to a question. You may need to use multiple methods to address all your important questions.
If you are evaluating process:
|Question||Possible Qualitative Methods||Possible Quantitative Methods|
|Did you engage with the right partners at the right time?||Interviews with partners and staff||Survey of partners|
|Did the measures you chose resonate with your audience?||Focus groups with sample of audience member||Surveys of audience members|
|Did your audience find your Web-based report?||Focus groups with sample of audience members||“Web analytics” to track usage of the Web site
Surveys of audience members
|Did you get the media attention you wanted?||Tracking of media mentions (by yourself or through a service)|
|Did your outreach efforts reach those who are less literate?||Interviews with outreach partners
Focus groups with less literate community members
|Note: A survey would probably not be appropriate for a less literate group, unless it was done by telephone.|
If you are evaluating results:
|Question||Possible Qualitative Methods||Possible Quantitative Methods|
|Did your audience understand the report?||Usability testing after the report has been issued||Survey of audience members including questions to test their knowledge of key facts and messages in the report|
|Did your audience use the report? How?||Focus groups with samples of audience members||Survey of audience members
Observed changes in enrollment in health plans or use of providers (very difficult)
|How did providers and plans respond to the report?||Interviews with individual plans and providers||Survey of plans or providers|
|Did the reports intensify quality improvement activities?||Interviews with individual plans and providers||Survey of plans or providers|
|With what results?||Changes in plan or provider performance over time on key metrics, including but not limited to those in your report|
A survey asks a systematic sample of a population a set of questions that they answer using a specified set of responses. The sample population could be community members (including those you hope to reach), people who actually use reports, or representatives of purchasers, providers, plans, or policymakers.
Surveys ask a series of questions that can be closed-ended (where a limited set of answers is provided for each question) or open-ended. The use of closed-ended questions means that survey results are quantifiable.
Surveys may be administered by mail, by telephone, in person, or over the Web. Some Web sites incorporate a survey “feedback” function that asks questions and solicits comments from site visitors.
What’s needed for surveys?
- A “sampling frame” from which you can choose a representative (i.e., random) sample.
- The survey instrument, preferably one that has gone through some initial testing.
- A cover letter or other form of invitation to motivate survey response.
- A way to distribute the survey (mail, telephone, or Web).
- A way to follow up with people who don’t respond to the survey.
- A system for creating and managing a database of survey responses.
- A plan and a method for analyzing the results.
- Either a “vendor” that will conduct the survey for you or staff skilled in survey design, administration, and analysis.
In a focus group, a small group of individuals spends 1 to 2 hours in a guided discussion of a small set of questions. The individuals typically have certain characteristics in common, but they may also be diverse on other characteristics.
Unlike questions on surveys, the questions asked in focus groups can be answered in any way that the participants choose. No predetermined answers are provided.
The interaction among participants and how they influence each other are both part of the “data” that is of interest. In some focus groups, participants complete a brief survey at the beginning to capture their demographic characteristics or other information. In others, participants respond to a stimulus provided by the moderator.
What’s needed for focus groups?
- Access to a pool of people from whom you can recruit focus group participants who fit your criteria (often provided by a private vendor that specializes in commercial and/or academic focus group research).
- A detailed moderator guide, with primarily or exclusively open-ended questions.
- A skilled moderator.
- A facility to hold the focus group session(s) that is convenient, neutral, and attractive without being too plush. Focus group firms often rent their facilities for this purpose.
- Any materials that you want to use to stimulate the groups’ responses.
- One or more ways to record the focus group (audiotape, videotape, or notes) and summarize or transcribe the conversation. Focus group firms can provide this service.
- A method to analyze the results of all your groups. This may include a qualitative data analysis software program.
- Staff who have skills in qualitative data analysis.