Application of Program Theory and Logic Model to Evaluate Immunization Disparity Program for Children under 3 Years

Abstract

With the outbreak of coronavirus disease 2019 (COVID-19) pandemic, health policymakers are adopting new policies regarding the issue of immunization disparities, especially for children in low-income communities of color who lack awareness and thereby access to vaccines. The purpose of this paper is to propose an evaluation framework using program theory-based evaluation approach and logic model to analyze and evaluate the immunization disparities in children aged 19-35 months. Data is collected from New York City department of Health and the U.S. Census Bureau for Northern Manhattan Start Right Coalition program which consists of 19,800 children, and the community-provider partnership includes 26 practices and 20 groups. Program theory is used to evaluate this community-based initiative with the logic model which is a visual depiction that illustrations the program theory to all stakeholders. The logic model highlights the resources, activities, outputs, outcomes, and impacts of the program to guide to planners and evaluators and to call attention to the inadequacies or flaws in the operational, implementation and service delivery process of the program in offering a new perspective on the program. This framework adds to the literature on evaluations of immunization disparities in determining whether evaluators can definitively attribute positive immunization outcomes in the community to the program and conclude whether it has potential in expanding or duplicating it to other similar settings, especially in other rural areas of the United States, and abroad, where routine immunization equity gaps are wide due to income, racial and ethnic diversity, and language barrier.

INTRODUCTION

1. Overview of immunization disparity in children

  The ongoing global coronavirus disease 2019 (COVID-19) pandemic has brought attention to the continuing vaccination disparity in low-income neighborhoods in a wide range of health policy discussions, especially for children aged 19–35 months. Under COVID19 lockdowns along with inconsistent health service delivery, the World Health Organization and the United Nations International Children’s Emergency Fund reported a greater decline in the overall number of children who failed to receive immunization [1]. Those who fail to receive full immunization on time are more susceptible in facing serious infections and illnesses such as hepatitis, tuberculosis, whooping cough, and diphtheria that are found to be vaccine preventable. Increased rates of unimmunized population that are underserved not only put themselves at risk but can have a deadly impact on the entire population as it drives down the overall immunity level [2].
  Moreover, children in communities of color face severe immunization disparities. Racial and ethnic disparities persist because families of color tend to be less financially stable, with limited access to healthcare education that are fragmented and low in quality. There is also a higher proportion of immigrant families who are usually undocumented and face language barriers in these communities. The Northern Manhattan area in New York City (NYC), particularly the neighborhoods of Washington Heights and Harlem, is characterized by these qualities and is among the most underserved and disadvantaged in the city and in the United States [3]. About two thirds of the families in these neighborhoods have incomes 200% below the poverty level, and a third of them receive support from the government [4]. In 2000, 52% of the population in these neighborhoods were Hispanics and 38% were African Americans. But with gentrification and racial residential segregation, the demographics have changed over time with Central Harlem being majority of 54% African American, Washington Heights consisting of 68% Hispanics, and East Harlem with 43% Hispanics, which predominantly paints a community of color being the majority group in 2021 [5]. In 2021, a significant group of residents in these communities were foreign-born in Washington Heights with the rate of 44%, Central Harlem 21%, and East Harlem 20%, respectively, with most of them from the Dominican Republic, West Africa, and other Latin American countries [6]. A recent study highlights how gentrification process has pressured the low-income class, primarily Black and/or Hispanic residents, to be racially “segregated” and “displaced” out of their homes which can have negative racial disparities in health and education, highlighting Harlem as an example [7].
  Even though there has been remarkable progress in childhood immunization coverage nationwide, communities of color still have coverage rates 5% to 15% below the national average [8] and recent findings report that the vaccination rate among children aged 19–35 months continue to be only 72.8% which failed to reach the 90% target set out by Healthy People 2020 initiative [9]. Factors such as community of color, income under poverty line and low education were significant indicators that lead to failed vaccination among children [10]. In the past, several strategies and programs have been implemented to reduce immunization disparities such as changes in clinic operation processes, outreach to patients’ families through telephone calls or mailings, community health worker visits, collaborative community approach, and provider incentives. The findings show that community health worker visits and other outreach methods for families seemed to be most effective in increasing immunization rate [11].

2. Programs to reduce immunization gaps and their limitations

  To encourage immunization among children under 3 years, the US government has implemented various pilot programs and projects in different cities. For example, the Northern Manhattan Start Right Coalition was a program aimed at increasing immunization access to children in communities of color where racial and ethnic disparities persist. The program worked on a community-provider partnership model to foster provider knowledge and accountability, practice improvements, and community outreach. The purpose of this paper is to propose an evaluation framework using program theory-based evaluation approach and logic model to analyze and evaluate the this program. Program theory, also known as theory of change or theory of action explains why and how a program is expected to work and is often used to evaluate a community-based initiatives with multiple partners and stakeholders [12]. The logic model, widely used in community-engagement projects in public health, is used as a visual depiction that illustrates the program theory to help all stakeholders understand the interrelationship between inputs and outputs. Past evaluation methods present inadequacies and flaws as they did not use the appropriate counterfactual and did not address threats to internal validity adequately. The results of this program were compared with the National Immunization Survey (NIS) coverage rates, which are population-based rates obtained through a random digit dialed telephone survey, for the US and NYC children. The coverage rates for Washington Heights were compared to rates for Hispanic children in the NIS survey; those of children in Harlem were compared to coverage for African American children in the NIS survey. One-sample t-tests and chi-square tests were used to assess significant differences. The evaluation concluded that immunization coverage increased at faster rates in Northern Manhattan compared to NYC and the United States, due to the program [13]. The counterfactual in this case is the NIS coverage rates obtained from samples representative of the general US and NYC populations. Children in all of the United States or NYC are inherently different from the target population in Northern Manhattan in terms of race, socioeconomic status, immigrant status and more. It is not valid to compare the three populations for this evaluation because the results would be biased due to internal and external factors. The design also does not address the effects of maturation and outside effects such other programs with similar goals that might be in process in other part of the United States or NYC. Therefore, this paper proposes a different design with a different and more appropriate counterfactual using the Program theory and logic model.
  The findings from the evaluation will have policy relevance as it is essential to make evidence-based decision in public health policy implementation. It will help disadvantaged families and children in Northern Manhattan and in similar areas where vaccine equity exists to get the appropriate intervention in order to improve immunization coverage. Therefore, this proposal will detail the design and process the evaluation will undertake to answer the hypotheses and following research questions as follows: (1) Do participants of the program have more children under age 3 who are immunized for diphtheria-tetanus-pertussis (DTap) than the comparison group 1 year after the beginning of the program. (2) Are providers who participate in the community-provider partnership more likely to implement centralized immunization policies than similar providers who do not participate in the program 1 year after the beginning of the program. (3) Are families in the program more likely to have access to bilingual immunization education materials than families in the comparison group 1 year after the beginning of the program?

METHODS

1. Study population

  This program serves virtually all children younger than 3 years in the neighborhoods of Harlem and Washington Heights, which totals 19,800 children, and the community-provider partnership includes 26 practices and 20 community groups.

2. Program theory and logic model

  Program theory explains and provides a detailed roadmap as to how a program is expected to work and is often used to evaluate a community-based initiatives with multiple partners and stakeholders. Logic models are often used as valuable tools in the field of public health and health promotion as it visually depicts the interrelationships of all the inputs with outcomes that are action-based and are measurable [14]. The logic model applied to the immunization disparity program for children under 3 years is shown in Figure 1.

 

RESULTS

1. Process evaluation

  Process evaluation will be conducted with information and feedback provided by program staff prior to and post the program. This step is critical in ensuring that the program exists and has been implemented the way it was planned and designed. For example, if the program was unable to engage enough community members, provide training to outreach workers, or distribute enough bilingual immunization outreach materials with the relevant languages, it would be possible to track these shortcomings early on and make adjustments so that the target population receives the appropriate services. It is critical to take note of whether all workshops at all levels including nurses, providers, outreach workers and program staff adhere to the guidelines as proposed. Other criteria such as recruitment for community partnerships, review of documents such as best practice immunization report cards, and the duration of workshop should be recorded. Moreover, as parents play an important part in the study, it is important to actively engage with the parents to deepen the level of understanding and shift in attitude of the parents as the program continues.
  The process evaluation will take place in the beginning, middle, and towards the end of the year-long program. Some of the methods in use are the following: interviews with program staff/nurses/providers and interviews or focus groups with parents to collect information on the effectiveness of the program. Level of engagement by the staff and providers, change in attitude, and any barriers to access to immunization will be noted. The researchers will attend or participate in workshops for training program staff/nurses/providers/outreach workers and will review bilingual immunization outreach materials. Finally, analysis of the findings from the process evaluation and improvements on recommendations on how the current and similar future programs will be presented.

2. Design

  This paper proposes a quasi-experimental design, using program theory-based evaluation approach with the use of logic model with a program group and a comparison group for the evaluation. The program group will be children, parents, and providers in Northern Manhattan, and the comparison group will be a neighborhood that is located in NYC and comparable to the Northern Manhattan in terms of demographics. The Jackson Heights neighborhood in the Queens borough of NYC represents similar demographic characteristics and make up of immigrant communities as the Washington Heights and Harlem [8]. In this design, this program for increasing immunization coverage will be provided to all families with children under age 3 in the Northern Manhattan, while the families in the Jackson Heights will not receive the program since they are the comparison group. The program community will practice the community-provider partnership model while the comparison community will not. Data regarding DTap immunization rates for children under age 3, implementation of centralized immunization policies, and availability of bilingual immunization education materials to families before the beginning of the program and 1 year after the beginning of the program will be collected. This design was selected as it allows comparison of children’s immunization results between the group that received the program and the one that did not, before and after the program. It also helps to compare results between the two groups in terms of provider knowledge and accountability, and access to bilingual immunization education materials. The design handles some threats to internal validity such as maturation because there are no existing maturational trends in the program and comparison groups. If there are, both groups would be on the same maturation path. The design also controls for the effects of attrition because of the existence of a comparison group. Both groups are prone to similar rates and reasons for attrition. There is little concern for high rates of attrition because the sample size in both groups is large enough to address this issue. The design also addresses the effects of contamination because the program and comparison group are geographically far apart from each other. Instrumentation is also addressed because any changes in the way secondary data is coded pre/posttest would affect both groups similarly. Testing effect and regression to the mean do not apply to this design because data is obtained from secondary sources and the survey questions are not mainly about perceptions.
  Although this design manages outside effects such as citywide policy changes, it does not handle the effects of other campaigns and similar programs that the comparison group might be exposed to during the course of the program. Because the individuals in the program and comparison group are not assigned randomly, this design does not manage a major threat to internal validity, which is selection bias. In addition, participation in the evaluation is voluntary. Although using a true experiment design was considered, where individuals are assigned into a program or control group randomly which would address selection bias, it was rejected because the services of the program are meant to be disseminated to everyone in the target population. Denying the services to some individuals who want to participate would be deemed unethical. It would also be logistically challenging because the program is meant to not solely evaluate but also engage providers, community organizations and families. This program and evaluation will have external validity only to populations or neighborhoods with similar demographic makeup as the current target population. The outcomes of this evaluation will only be applicable to neighborhoods that have a vast majority of families of color and immigrant families that have low incomes.

3. Sample

  The program’s target population is communities of color and their children who are under 3 years old. The program also involves and plans to see outcomes with providers and community organizations in these communities. The first sample for the program group will be children within Harlem and Washington Heights who are younger than 3 years. The population data can be obtained from the NYC Department of Health and the Census Bureau and totals about 19,800 children. In this case, the target population and sampling frame are the same. The comparison group will be all children who are younger than 3 years in the Jackson Heights. The two neighborhoods are comparable in terms of demographic characteristics and income level as described above. Since this is secondary data from official sources, information on all, if not the majority, of the children in both neighborhoods is expected to be obtained. The final analysis will compare the rate of immunization in both Northern Manhattan and Jackson Heights before and 1 year after the beginning of the program.
  The second group of samples will be local providers in the program neighborhood who will be actively engaging in the community-provider partnership model and implementing centralized immunization policies. It is expected to have 50 providers from the 26 practices who are involved in the program. Another 50 providers will be invited from Jackson Heights to be part of the evaluation and participate in pre- and post-surveys, ensuring that the providers in the comparison group are from similar facilities as the ones in the program. It is expected an almost full participation from the program group since they are part of this partnership; however, the researchers might experience a lower response and retention rate from the comparison group.
  Third, to assess and compare the level of access to bilingual immunization education materials in both neighborhoods 100 households from each of the program and comparison neighborhoods will be randomly selected. Sample selection will be based on a random lottery from the existing individual home addresses in those neighborhoods. Researchers will administer a survey to these households before the beginning of the program, which will serve as a baseline assessment. For this survey, the first question will ask whether there are any children under 3 years in the household. This will allow researchers to pick only those surveys with a positive answer to this question for the analysis because the program is for families with children under 3 years. Same individuals who are selected for the baseline assessment will be invited to take the same survey 1 year after the beginning of the program. In order to increase the response rate and retention rate among these randomly selected households, the program will offer parents physical consultation to for the participating parent and guardians who complete the survey to secure external validity for the target population.

4. Measures

  For the first research question of measuring immunization rates, the nominal level of measurement will be used to assess significant differences between immunized and unimmunized children in the program and comparison group. They will access data from the local immunization centers and the health departments on how many children under age 3 have been immunized for DTap within the past year for both the program and control group. Then this can be compared with the number of immunizations in both groups to see if there are more children in the program group who are immunized compared to the comparison group. Finally, the researchers will conduct a chi-square statistic to test whether the difference in immunization rates between the two groups is statistically significant. Since the data for this measurement are obtained from secondary sources, the evaluators will report on data collection, coding, missing data, quality control and data entry methods used by the local immunization centers and the health departments as they will obtain data to secure the reliability of these responses. In addition, to establish face validity of the measure that the results reported are only for DTap and only within the past year. Researchers will further confirm as to how many follow-up immunization visits for DTap is considered a sufficient measure by the public health and medical community in order to establish content validity.
  The second question of measuring provider knowledge and accountability tests whether providers who participate in the community-provider partnership are more likely to implement centralized immunization policies put forth by the Centers for Disease Control and Prevention (CDC) compared to providers who do not participate in the program. To do this, a nominal level of measurement where the providers in the program and comparison group whether or not they are implementing the policies. CDC Guideline and official documentation that proves whether these facilities are implementing the policies will be obtained. Chi-square statistical test can be used to confirm whether there is correlation between participating in the program and implementing centralized immunization policies and whether that correlation is statistically significant. Since data for this measurement are obtained from both primary and secondary sources, researchers will establish internal consistency and conduct a test-retest reliability in order to assess the reliability of the survey used to collect primary data. In order to assess the reliability of the secondary data, researchers will access the facility documentation to analyze how data about new policies are documented, when they are documented and implemented, and who applies them. In addition, criterion validity will be established in order to assess whether the results measure with the centralized immunization policies recommended by the CDC. For example, the CDC recommends that DTap is routinely administered at 2, 4, and 6 months, and at 15 through 18 months [15].
  Regarding the last question of measuring the availability of bilingual education materials for families, ordinal level of measurement—a Likert Scale asking access to access to bilingual immunization education materials to both the program and comparison group, before the beginning of the program and 1 year after the program will be used. Questions are such as “Do you think that having free access to bilingual immunization brochures translated to your native language is helpful?” and “Do you think that having immunization education materials that are translated to your native language help you become more aware of the benefits of vaccines?” The responses will be averaged for both groups and use the t-test to compare the group means and analyze whether there is correlation between participating in the program and access to bilingual immunization education materials and whether that correlation is statistically significant.
  In order to assess the reliability of this measurement, the researchers will establish internal consistency and conduct a test-retest reliability to measure how reproducible these tests and results are. Individuals from the program and control group will complete the survey once, and then the same survey will be administered to the same people after a month to see how stable their responses are. Then the correlation coefficient of the two sets of responses will be calculated to compare them. If the calculated correlation coefficient is at least 0.7, it is fairly reasonable to conclude that the survey responses are reasonably consistent from one point to another. In addition, to establish face validity, untrained individuals will be invited to check whether they think the items in this survey make sense and ask the basic idea of the question they are supposed to ask. To establish content validity, a set of vaccine experts and the low-income immigrants will be invited to assess a subjective measure of the appropriateness of the survey. Program staff, community workers, and healthcare providers would be reliable experts for the topic this evaluation is trying to measure. Even though this method is not a scientific measure of a survey instrument’s accuracy, it provides a good background for a methodologically rigorous assessment of its validity.

5. Procedures

  During the first month, researchers will reach out to practices and community organizations in Harlem and Washington Heights to establish a community-provider partnership. They will have first meeting with these practices and organizations in order to communicate the goals and establish the logic model of the program. This meeting will address what is expected of the practices and the community organizations and what they should achieve. The participants will also receive their first trainings and workshops. Moreover, official information on the number of and list of families with children under age 3 in both the program and comparison neighborhoods will be information from the Census Bureau, Department of Health and Medicaid data will be obtained.
  During the same time, researchers will also do a baseline assessment of the percentage of children under 3 years of age are immunized for DTap, whether providers implement centralized immunization policies put forth by the CDC, and whether families have access to bilingual immunization education materials for both the program and comparison neighborhoods. Researchers will also randomly select 100 households from each of the program and comparison neighborhood and administer the surveys regarding bilingual immunization education materials to these households. These assessments will rely upon data from both primary and secondary sources (such as the health department and the facilities) and use methods described in the Measures section above. The assessment will be done for both the program and comparison group.
  Process evaluation will be conducted during the beginning, middle, and end of the program. This will ensure that the program is being implemented as planned and to check whether the target population is getting access to the services provided by the program. Researchers will keep track of the number of individual community outreach workers, the number of household visits, the number of immunization workshops held for nurses and office staff, and the number of outreach materials and flyers distributed during each of these checkpoints. This process evaluation helps to collect information about possible barriers in the implementation of the program from several perspectives and improve them accordingly.
  Finally, the researchers will collect outcome data for the program and comparison group 1 year after the beginning of the program by measuring the rate of immunization for DTap in children under 3 years, and whether providers implement centralized immunization policies put forth by the CDC, and whether families have access to bilingual immunization education materials. In response to the last question, the researchers will administer the same survey to the same randomly selected households that completed the baseline assessments and are still willing to participate. The measures used at this stage of the evaluation will be the same as the ones used for the baseline assessment explained above. Then they will compare the pre/post program results for the program group and the comparison group. A final report will have details of the analysis findings by the end of the following year, in accordance with the CDC guideline.

6. Analysis report

  Three key outcome measures are studied in this evaluation in addressing the research questions. The first hypothesis is that participating in the program is associated with increase in DTap immunization rates for children under 3 years. To answer this question, DTap rates for children under 3 years from a secondary data provided by the Department of Health for both intervention and comparison group will be collected to calculate the overall increase from pre to post evaluation year. Since this is a nominal level of measurement, a Pearson’s chi-square test can check whether the difference in immunization rates between the two groups during this time is statistically significant. Similarly, the same chi-square test can be used to compare the counts of categorical responses between the two independent groups for the second question of whether providers who participate in the community-provider partnership are more likely to implement centralized immunization policies recommended by the CDC.
  Finally, the researchers will then conduct a paired t-test on the data obtained from the surveys to analyze the rate of access to bilingual immunization education materials. This allows to compare the means of the program and comparison group and determine if the average of the program group increases or decreases over the year in relation to the comparison group. Moreover, it can be determined if this difference is statistically significant. The analysis will also shed light on which demographic groups are more likely to use the services and whether the immunization education materials are translated to the appropriate languages reflecting the characteristics of the target population.

Conclusion

  The evaluation results will have implications for policy and for various relevant stakeholders including funders and intended beneficiaries. If the evaluation finds positive results, the healthcare providers along with program planners and staff can continue inputting more time and resources into the program where appropriate. This finding also encourages policymakers to implement this project in other similar neighborhoods in NYC and other cities where child immunization disparity persists, especially in rural areas that are often isolated. If the evaluation finds both positive and negative results, it will be a signal researchers that the program theory should be altered to better serve the project goals. This will be supplemented with findings from the process evaluation. Finally, non-findings will point out the need for a different intervention. Most importantly, evaluation results will help the intended beneficiaries get the appropriate intervention in a timely manner. This program saw merit and success in reducing immunization gap over the time in terms of the overall numerical report. However, it is important to conduct a process and outcome evaluation of this program as process evaluation will allow planners and evaluators to view the program in a new light and highlight any inadequacies or flaws in the operational, implementation and service delivery process of the program.
  The proposed evaluation is limited by the fact that it is not a true experiment: one that uses random assignment of individuals into program and control group. Due to this reason, the current evaluation design cannot fully address the effects of selection bias. Some families might participate in the program more than others because they need the services most. Therefore, this might somewhat bias the results of this evaluation. However, a true experiment design was considered and rejected because it would be unfair and unethical to deny services to some families within the target population. Since the intervention has several educational components, using random assignment within one neighborhood might lead to contamination. Although it would be viable to evaluate the outcomes of the program with regards to immigrant families, particularly on access to bilingual immunization education materials, it would be challenging to define the term “immigrant” because it could have several aspects such as possession of legal documents and duration of stay in the country. Some individuals might also not want to disclose their immigration status, and this might lead to false data or decreased sample size. Despite these limitations, this proposed framework adds to the literature on evaluations of immunization disparities in determining whether evaluators can definitively attribute positive immunization outcomes in the community to the program and conclude whether it has potential in expanding or duplicating it to other similar settings, especially in other rural areas of the United States, and abroad, where routine immunization disparity is high due to the factors discussed in this paper. It is important to note that the findings of this research is also relevant in the Korean context. Although Korea has historically been an ethnically and culturally homogeneous country, the number of immigrants along with international marriages have grown rapidly in the last decade, especially in rural and isolated regions due to labor shortage and shrinking population. The government is heavily investing various social integration programs as the influx of immigration will only continue to rise. As this group mostly live in rural areas with limited access to health institutions with high language barrier and low health literacy, previous studies have highlighted the importance of implementing welfare policies and programs to reduce health inequalities for these communities [16]. The framework proposed in this study can be used as an effective public health intervention in Korea that can protect the children in these communities from preventable infections.

 

CONFLICT OF INTEREST

  No potential conflict of interest relevant to this article was reported.

 

ORCID

Jee In Chung: https://orcid.org/0000-0001-6647-2244