CHAPTER THREE
Methodology
3.1. Introduction
Mentoring programs have become a popular strategy for managing the needs of young people who are considered at-risk by a set of distinct criteria. Thus, a number of schools and social service agencies are presently conducting programs for mentoring local youth, such as Fresh Start, discussed in Chapter Two. The research proposed here is twofold. First, using a descriptive approach that primarily draws on quantitative data the study seeks to examine the impact of a specific mentoring program – Fresh Start. Second, the study aims to examine the context and focus of the Fresh Start mentoring program on at-risk youth. This chapter outlines gaps in current knowledge concerning the evaluation of mentoring programs and presents the aims of the study, study design, and methods for the collection, storage, retrieval, analysis and presentation of data.
3.2. Evaluating the effectiveness of mentoring programs
In a mentoring arrangement, adults usually take the role of a parent or advisor and are paired with young people to establish supportive and trusting relationships. Aside from the fact that mentoring programs enable community development and cooperation, they can be customized based on the students’ personality and learning capabilities (1997). The development of an effective mentoring program for at-risk youth requires explicit attention to the primary needs and concerns related to today’s disadvantaged youth population. It is important that the research instruments and analyses are conducted in ways that promote a balanced agenda and the most appropriate course of action for at-risk youth.
3.2.1. Rationale for Evaluating Mentoring Programs
In the United States, the increased awareness of the availability of various mentoring programs has led to their rapid growth and popularity. These mentoring programs are focused on the establishment of a relationship between a troubled youngster and a caring adult. This relationship is nurtured by means of spending quality time together and providing substantial support and guidance. In general, the primary objective of these mentoring programs is to help young individuals overcome the difficulties of life (2002). Since so many children and young adults experience tremendous challenges in their efforts to maintain some sense of stability in their lives, it is not surprising that they are likely to respond positively to mentoring relationships, especially when they are carefully planned and executed. The development of these mentoring associations is of critical importance, and an element of chemistry must be present in order to reap benefits from them. Evaluation is an important component in measuring social impact and in identifying challenges to successful implementation.
3.2.2. Evaluating Efficacy in Mentoring Programs
While mentoring programs may have clear objectives and established approaches regarding youth concerns, the efficacy of these programs remains in question. While some research emphasizes the positive effects of mentoring programs on at-risk youth (1997;1995;1993), a comprehensive meta-analysis of a range of programs suggests that knowledge about the effects of mentoring programs is still in its infancy stage (2002). In addition, such programs may accrue only modest benefits to participants (2002). The greatest benefits for at-risk youth are observed where ‘best practices’ are theoretically and empirically derived and where there are established relationships between mentors and mentees (2002). There are considerable advantages in conducting a quantitative research study that provides a statistical basis for examination and analysis of mentoring programs at-risk youth. With the appropriate research instrument in place, there are many opportunities to identify the benefits of at-risk mentoring programs as well as their disadvantages and to determine the level of improvement that is required in developing these programs and their long-term outcomes.
3.3. Aims of the Study
The research proposed here is twofold. First, using a descriptive approach that primarily draws on quantitative data the study seeks to examine the impact of a specific mentoring program – Fresh Start. Second, the study aims to examine the context and focus of the Fresh Start mentoring program on at-risk youth. Within this program, the majority of at-risk youth come from single-parent homes. Typically, they manifest behavioral or emotional problems and lack the support necessary to handle developmental tasks successfully. As discussed in Chapter Two, once at-risk youth grow into adults, they are likely to be involved in chronic unemployment, divorce, substance abuse, physical and psychiatric problems, divorce as well as other forms of criminal activity (Patterson, DeBaryshe, & Ramsey, 1989: 331). There are considerable challenges involved in developing an effective mentoring program that will effectively reach and infiltrate the minds of at-risk youth, who are particularly vulnerable to difficult circumstances and have likely constructed defenses to manage the problems that they have experienced. Therefore, it is highly likely that the proposed research instruments will reach the intended audience, but not without experiencing some level of difficulty beforehand.
The Fresh Start program is designed to provide specific mentoring activities as needed for at-risk youth that possess various behavioral and academic problems. The mentoring relationship offers participants the opportunity to get to know another individual who provides much-needed leadership and guidance in managing the complexities of participants’ daily lives.
The proposed study contributes to the evaluation literature in relation to mentoring programs by examining the efficacy of a particular mentoring program and the effect it has on the academic and behavioral development of at-risk participants. Most importantly, the proposed study attempts to identify areas of strength and weakness within the Fresh Start program by defining key variables for consideration and application in promoting advanced outcomes for at-risk youth. Specifically, the study proposed here intends to examine the quality and efficacy of Fresh Start by asking the following:
1. Which factors significantly affect the impact of Fresh Start on at-risk youth, and how are these factors incorporated into the key programmatic elements?
2. What are the key changes in the behaviors of at-risk youth participants in the Fresh Start program as based upon its key objectives? How will it be determined that these objectives are operating successfully in order to improve the lives of participating at-risk youth?
3. How does Fresh Start provide a measurable impact on the academic and behavioral development of its at-risk participants? What measures will be created to determine the degree of significance of the Fresh Start program in the lives of its participants? Will these measures be evaluated on a continuous basis over a longer period of time, or will they be considered during a single evaluation?
4. Are there ways in which the delivery and impact of Fresh Start can be improved? Most importantly, what type of impact does this program represent for at-risk youth, and will criteria be developed that will evaluate the importance of this program over the long term? Is it possible that these measurements could be duplicated and applied to other programs for at-risk youth, and how will they be utilized?
Based on these research questions the project offers the following null hypothesis:
· Fresh Start has no significant impact on at-risk youth’s academic development.
· Fresh Start has no significant impact on at-risk youth’s behavior.
The objectives of the research include the following:
1. To determine if Fresh Start has a significant impact on at-risk youth in terms of their academic and behavioral development;
2. To determine which factors are most significant in Fresh Start’s impact; and
3. To formulate ways of enhancing the delivery and impact of Fresh Start.
Based upon these key principles, it is important to note that quantifiable analyses must be conducted by evaluating key variables and their significance in effectively promoting advanced outcomes for at-risk youth. For example, it is possible that some aspects of the Fresh Start program are more effective than others in promoting change and progress for at-risk youth, and therefore, those areas of weakness that have been defined must be reduced or eliminated altogether in favor of advancing the positive aspects of this program. Without identification of quantifiable measures, it is difficult to justify the program's success for its participants, thereby making its existence futile.
3.4. Challenges for Mentoring Program Evaluation
While research suggests that youths participating in mentoring programs tend to have higher self-esteem, higher grade point averages, better attendance, and fewer suspensions, because the focus of mentoring programs is diverse and the methodologies used to evaluate efficacy and effectiveness vary, it is often difficult to gauge the overall impact of mentoring programs on at-risk youth (2002). The cost of funding fully operational research usually exceeds the limited budget of a mentoring program; thus, many evaluations have relied exclusively on self-report information rather than on research instruments that offer validity and reliability. There are other problems. Many mentoring programs that do conduct evaluative research to determine the efficacy of their services do not employ methodologically sound techniques. For instance, many either fail to use control groups or do not utilize non-random assignment. They fail to account for issues such as the intensity of treatment and the quality of the mentoring contact, and the variations that both factors can create in the observed variables. These inconsistencies within the research lead to many weaknesses in program efficacy for at-risk youth, and by solely evaluating programs based upon self-report information, there is likelihood that those individuals requiring the greatest level of support and guidance will not necessarily gain the most from the program itself. Therefore, it is important for research to account for the potential long-term advantages of comprehensive evaluations that do not solely utilize the self-report mechanism.
In general, self-reporting mechanisms do not necessarily provide the most optimal level of results that are desired, due in part to potential bias against providing the most accurate responses to the questions being asked of them. Furthermore, there are other problems with utilizing self-reporting as the sole means of evaluation, including the lack of perspective that is offered by other individuals involved in these specific circumstances. By incorporating the perspectives provided by teachers and other individuals in the lives of at-risk youth, the potential exists to improve feedback and to advance the effectiveness of these programs in future years.
There are other challenges in researching the effectiveness of mentoring programs. It must be remembered that while different mentoring programs may have similar goals, their structures and procedures may be dramatically different. For instance, some mentoring programs offer services to specific student sub-groups. Recall, for example, that the organization 100 Black Men works specifically with African-American young men. Similarly, mentoring may be used to promote academic success, as in the case of HOSTS; to prevent delinquency; or to prepare students for professional success. Finally, mentoring programs vary in what they require of the volunteer mentors.
While some programs, such as 100 Black Men and Fresh Start require training, other organizations do not. Similarly, there are varying degrees of oversight of volunteer mentors, and different requirements about the frequency, intensity, and duration of mentor-mentee meetings. Problems with the delivery and impact of mentoring programs may be due to their dependency on donations and volunteers (2002). Specifically, mentoring programs that do not designate a specific sector of the population may experience problems in attempting to convey the real needs and significance of these programs to their participants and other supporters. In general, attempts to effectively conduct and promote these programs without a specific focus will lead to disaster, thereby potentially reducing the advancement of at-risk youth to the desired level. By developing a specific focus for mentoring, there is a greater likelihood for success in terms of youth outcomes and personal development.
The effectiveness of programs can also be compromised by the frequency of face-to-face contact between mentors and mentees. Some programs require meetings only once or twice a month, which may be insufficient to provide all the help and support at-risk youth need. For many at-risk youth, there is a greater need for relationship building on a more frequent basis, as participants must feel that their mentors are indeed committed to their advancement. Evaluation needs to account for these differences in meeting frequency.
The research proposed for this dissertation addresses specific concerns regarding mentoring program efficacy by conducting a quasi-experimental study of an existing intensive mentoring program, Fresh Start. Fresh Start has been selected as an appropriate mentoring program because it works specifically with an at-risk population; it has a well-implemented training program for mentors; and it has a well-developed focus on community involvement. The goals of Fresh Start are to develop self-esteem and self-discipline; increase school attendance; and prevent the onset of a delinquent lifestyle. These are the factors that will be the main focus of the study operationalized in the following ways:
· Data related to several of the predictor variables that are correlated to future delinquency will be obtained;
· Indices of self-esteem as obtained through structured questionnaires;
· Comparison of pre-and post mentoring program participation questions as presented via Child Behavior Checklist instruments;
· This study will also incorporate parent, teacher, and peer report data about participants’self-esteem, self-discipline, school attendance and number of delinquent acts. This study will assess youth involvement in a mentoring program over the course of a six-month period.
3.5 Methodological Approach for Quantitative Research
All primary data will be collected using the "Child Behavior Checklists” in the form of pre and post intervention interviews. The checklists were devised by (1991) routinely used to study a range of behaviors and academic achievement among young people.
Three standardized checklists will be utilized for the purposes of this research study, and they will be described in the following paragraphs. The Child Behavior Checklist for Children aged 6-18 (Appendix One) consists of a general set of questions regarding such topics as child personal interests, group activities, chores in the home environment, friendships, the degree of ability to get along well with others, academic performance, special educational requirements, academic problems, and specific concerns regarding the child. Most importantly, the questionnaire addresses a series of 113 items regarding various behaviors and/or physical ailments/habits that the child possesses from a parental perspective, all of which could contribute to the current circumstances that the child faces in their daily activities. This questionnaire is comprehensive, as it emphasizes behavioral aspects of a child’s current state, and it is anticipated that these responses will lead to new questions regarding the importance and overall effectiveness of mentoring programs.
The second questionnaire, the Youth Self-Report for Ages 11-18 (Appendix Two), asks similar questions to the previous questionnaire, with the significant difference that these responses are generated from the youth perspective, which may be significantly different from the responses generated on the Child Behavior Checklist. It is anticipated that responses from the youth perspective will be unique and distinct from all others in some respects, although some revelations provided on this questionnaire could be utilized for identifying areas of strength and weakness within mentoring programs. Youth responses are particularly important in providing the most accurate assessments of mentoring programs, although they only serve as one of the many perspectives that are sought in advancing these programs to a higher level.
Finally, the Teacher’s Report Form for Ages 6-18 (Appendix Three) provides another assessment of various behavioral aspects of at-risk youth, as well as their academic performance. Teachers are questioned about their knowledge of the student’s history and their personality, their aptitude test scores, any known disabilities or problems, and other related issues. Teachers play a critical role in identifying areas of strength and weakness, as well as opportunities for improvement over a period of time. Teachers are important determinants of the best course of actions for their students, and they possess the knowledge of individual student behaviors that is necessary to improve their performance on both a personal and an academic level. Finally, teachers offer the best assessment of the necessity for participation in youth mentoring programs, and suggest the frequency of meetings and the severity of the problems that might exist for students.
Survey questionnaires, generated from secondary data, comprised of recent literature related to mentoring, mentoring programs and at-risk youth, will supplement data from the pre-post questionnaires and be distributed to the children’s parents or primary caregivers. Specifically, the questionnaire instruments will offer a greater understanding of the challenges that at-risk youth face in their daily activities. With a comparison across the three questionnaires, it is possible to obtain a well-rounded perspective of the current needs of at-risk students, including personality traits, behavioral characteristics, academic performance, educational motivations, and family history, amongst others. Although the perceptions of individuals might be diverse, this will offer a greater understanding of the problems that are faced by youth in attempting to cope with the external environment and their disadvantaged backgrounds.
Using quantitative analysis, each of the questionnaires will be evaluated based upon the responses that are given and will be scored based upon the category of the individual completing the form (e.g. parent, teacher). In addition, a cross-examination of questionnaires will be conducted for each student, with all three questionnaires being evaluated for their significance in comparison to each other. It is important for this cross-examination to take place as a means of identifying areas of weakness and decline within student morale and performance, so that the appropriate level of intervention will be identified and implemented whenever possible. There is an important distinction to be made regarding the differences amongst the three questionnaires. Although each questionnaire is designed to ask a similar set of questions, the diversity in responses for the same individuals could indicate that there are significant perceptive differences amongst individuals completing these surveys, which could reduce the level of support and intervention that is provided for at-risk youth. It is possible that mentoring programs may hold a different level of significance for each individual, and that some will experience significant success with these programs, while others will not gain any benefits. However, gaining a well-rounded perspective regarding mentoring programs is the only method for confirming that these programs are effective for students in need.
3.6 Researcher’s Key Scientific Assumptions
In this research proposal, the researcher aims to contribute to the current literature on mentoring programs for at-risk youth by means of determining the impact of one particular program on the academic and behavioral development of participants. Establishing effective interventions such as mentoring programs that strengthen and support young people is essential to their overall development and academic performance. Evaluating a current program will provide feedback to those involved and affected by it and promote constant development of the program so as to better serve the youngsters considered at-risk. Helping children to develop and discover themselves through mentoring programs will also enable them to live better lives, become more confident and productive.
Hence, the benefits of this study are not only short term but will also contribute to the long-term development of mentoring program evaluation. It is anticipated that mentoring activities that are conducted at this stage of life will be beneficial to students on a long-term basis, and that as they experience other areas of need throughout their lives, they will gain insights into the benefits of establishing effective relationships with mentors and other individuals in situations as they grow older. Therefore, the benefits of such programs are highly significant not only during the phase in which they are conducted, but throughout the entire life cycle.
3.7 Unit of analysis, locating and selecting research participants
The primary unit of sampling and analysis will be youth within the age range of 13 to 17 years who are considered at-risk. For the selected youth, their parents and mentors will also be recruited and their questionnaires will be analyzed. A random sample will be selected from enrollees of a mentoring school located within Los Angeles County California by the method of simple random sampling, in which each subject of the population has equal probability of being selected. The school draws participants from three geographic areas that are characterized by socio-economic disadvantage. Census data (2000) indicate that these ethnically diverse counties have a higher than national average percentage of families living below the poverty threshold, lower than national average median income and high unemployment. A list of potential participants - all new mentees - will be provided by mentors associated with the Fresh Start program run by the Quantum Leap Consulting Agency. Each new mentee will be assigned an identification number. Typically there are 175-200 mentees in a new cohort. There are generally more females than males (approximately 4:1) in a cohort, which has an age range of 13-18 years. About 100 mentees will be randomly selected from the list using a random number table. Then the Quantum Leap Staff will contact the selected mentees and their parents for an initial orientation that addresses the program details (meeting, dates, times locations). The mentoring process will last for six months. In order to assess the significant changes that take place after the mentoring process, parents of youth participants who attend the mentoring program will be given a survey questionnaire for evaluation. The study will be conducted in this manner as a means of promoting the likelihood of change and advanced support for improving total outcomes, from academic performance to personal agendas. At-risk youth face many precarious circumstances during these years, and their confusion and frustration with school and other activities plays a significant role in their personal and academic advancement. Therefore, new alternatives must be utilized in order to provide them with even greater opportunities for obtaining positive outcomes.
3.8 Data Collection
The data that will be gathered for this study will be primarily derived from the Child Behavior Checklists and a supplementary questionnaire. These structured questionnaires will generate responses from three groups: parents, youth and teachers. The information contained in these checklists will serve as the basis for comparison of any significant changes acquired by the children after attending the six-month mentoring program. As mentioned previously, that data resulting from the three separate questionnaires will be compared in order to determine the effectiveness of mentoring programs for at-risk youth versus those that have not participated in such programs.
In developing the statistical analyses of the completed questionnaires, patterns are likely to be recognized as critical to the overall success of the program and its outcomes. Some of these patterns might include - but are not limited to - poor academic performance, a variety of behavioral problems, and family issues, amongst others. It is highly likely that one or more of these problems has served as the primary contributing factor in the lives of at-risk youth, and that as mentoring programs get underway, they will ultimately provide additional sources of support, guidance, and knowledge for disadvantaged students in need. Therefore, mentoring programs are designed with these specific needs in mind, as students will experience the benefits of such programs through relationship development and effective outcomes over a period of time.
These questionnaires contain questions pertaining to the significant academic and behavioral developments that the youth acquired after attending the six-month mentoring program. The respondents will grade each statement in the survey-questionnaire using a defined measurement scale. For the behavioral section of the questionnaire, respondents will be asked to answer in one of the following ways:
0 = Not True; 1 = Somewhat or Sometimes True; and 2 = Very True or Often True.
Other sections of the questionnaire require specific written responses as well as individual information regarding such criteria as grades and test scores. It is evident that the anticipated results of the study will offer some general insights into the specific behaviors that lead to disadvantaged circumstances.
A five-item Likert Scale will be used, following the model proposed by (2000) and will provide respondents five choices for answers. The researcher opted to use the questionnaire as a tool since the rules of construction are easy to follow. Moreover, copies of the questionnaire could reach a considerable number of respondents either by mail or by personal distribution. Generally, responses to a questionnaire are objectified and standardized and these make tabulation easy. Finally, the questionnaire instrument serves as a strong method of identifying areas of strength and weakness amongst the participant population, as well as typical and atypical patterns of behavior that might exist. It is important to note that these questionnaires will also serve as indicators of the program’s influence on at-risk participants, as based upon the responses that are generated.
3.9 Capturing, Storing, Retrieving and Safeguarding Data
The data gathered from the pre and post intervention interview will be placed in the Child Behavior Checklist form, which will be accomplished by the staff members of the mentoring agency. After the six-month mentoring program, the structured questionnaire will then be personally distributed by the researcher to the parents of participating youth. In order to safeguard the data contained in the survey form, the researcher will personally retrieve the accomplished questionnaire. Maintaining confidentiality regarding all personal information and questionnaire data is of the utmost importance in obtaining positive and accurate responses to the questionnaires. Therefore, by ensuring that all responses will remain confidential, it is possible that all information will be reliable for testing and evaluation purposes, and that the potential effectiveness of the mentoring program will not be questioned by falsified data or a weak research instrument.
3.10 Data Analysis
The pre-intervention data from the Child Behavior Checklist will be analyzed first. The information of the post-intervention data gathered from the children and parents will then be compared to the data from their pre-intervention data. The analysis of this data will enable the researcher to determine the effectiveness of the mentoring program. The questionnaire, on the other hand, will be analyzed by assigning values to participants’ responses. The equivalent weights to be used for the answers are shown in Figure 1.3:
Figure 1.3: Weighted Scale for pre-intervention data from the Child Behavior Checklist
Range Interpretation
4.50 – 5.00 Strongly Agree
3.50 – 4.00 Agree
2.50 – 3.49 Uncertain
1.50 – 2.49 Disagree
0.00 – 1.49 Strongly Disagree
A detailed analysis will be conducted using the Assessment Data Manager Windows Software, which is aligned with the Achenbach System of Empirically Based Assessment that frames the behavior checklists used to gather data. This particular program is desirable because it enables rapid data entry and the ability to score and compare data streams obtained from parent, teacher, or peer reports. Comparison of responses and scores will be possible for up to eight forms per individual. For the purposes of this research study, since there will only be three questionnaires scored per individual, it is possible to obtain the desired results with fewer opportunities for errors in the scoring process. These findings will then be evaluated for their contribution and value in promoting effective outcomes for current and future mentoring programs.
3.11 Data Presentation
The quantitative findings derived from the questionnaire will be presented in text and graphic formats for easier analysis, comprehension and for the identification of patterns and unique circumstances that might lead to complex results. It is important for discussion and dissemination purposes to develop results that are easily quantifiable, easy to read, and that can be evaluated without considerable difficulty. The results as presented in a report format will allow other experts as well as novices to better understand how mentoring programs provide considerable support in advancing the overall outcomes of at-risk youth.
3.12 Ethical Considerations
As this proposed research requires the participation of children, parents and teachers there are a number of ethical and conduct considerations that are worth mentioning. According to Mouton (2001), the researcher should ensure his or her compliance to four main rules relevant to the rights of the respondents or participants. First, the researcher should recognize the respondents’ right to privacy, which covers their right to refuse to partake in the research process. A second aspect to consider is participants’ right to confidentiality and anonymity. The researcher should also ensure that the participants are able to receive the full disclosure of the research. More importantly, the respondents should not be placed in situations that can possibly harm them physically, psychologically or emotionally. Specifically, confidentiality is of critical importance in any research study, particularly when specific identifiers of personal information are included, such as name and address. For this research study, names as well as academic performance are discussed at length, and therefore, it is necessary that these items must be protected at all costs in order to safeguard the integrity of participants and their families. When minors are involved in a research study, it is particularly important that the identities of all study participants are protected. Furthermore, when academic information is involved such as grades or test scores, a school’s integrity and confidentiality must also be protected.
0 comments:
Post a Comment