Value-Creation Evaluation Framework for Evaluating NSF Advanced Technological Education Projects

Introduction 

NSF’s ATE Program is 30 years old, having been created by the Scientific and Advanced-Technology ACT of 1992 with initial federal budget appropriation in 1993. The mission of NSF’s Advanced Technology Education Program, as stated in the current ATE solicitation (NSF 21-598), is: 

“With a focus on two-year Institutions of Higher Education (IHEs), the Advanced Technological Education (ATE) program supports the education of technicians for the high-technology fields that drive our nation’s economy [1]”. 

Each year, NSF’s ATE Program receives grant proposals from eligible organizations, primarily two-year colleges, that educate the nation’s technical workforce. Submitted proposals undergo a peer review process; NSF Program Officers review them and, if recommended for funding, a final review by the NSF Division of Grants and Agreements. A requirement for all ATE-funded work is that the research be evaluated. Project descriptions must include a subsection titled “Evaluation Plan,” and the ATE Solicitation specifies the type of information necessary to adequately describe the plan. The ATE Solicitation also includes a directive that “there must be clear alignment between the evaluation plan and the project’s intended outcomes, activities, and deliverables” [1]. 

To a Principal Investigator and Project Team who are focused on their project vision and the resources, activities, and personnel that will be required to achieve project goals and objectives, creating an evaluation plan can be a daunting task. An experienced ATE evaluator can assist the project team in developing an effective evaluation plan and meeting the proposal requirements outlined in the Solicitation. 

Methods 

Laying the Groundwork 

This article assumes that the Principal Investigator (PI) has sought and found an evaluator for the proposed project. The first task facing the evaluator is to communicate the importance of evaluation in ATE grants. The value of including evaluation goes well beyond the NSF-imposed requirement for evaluation. The evaluator can describe the ways that evaluation will help the PI implement and execute a successful project. However, taking a longer view of grants, the ATE Solicitation states that if a subsequent proposal is submitted, the first section of the second proposal must be entitled “Results from Prior NSF Support” [1].  In this section, the PI must show that the prior proposal produced significant results. In terms of Intellectual Merit and Broader Impacts, according to the two NSF Merit Review criteria, a proposal must show how the prior work improved technician education and impacted the technical workforce, thus laying the groundwork for a new proposal. Evaluation data is used to support claims made, and evaluation findings will point to the next steps needed to further technician education. 

A potential barrier in working together and to effective evaluation planning and implementation is communication. The PI and evaluator will often speak using different vocabularies. The PI, usually a technical faculty member, will use a vocabulary linked to their technical discipline, while the evaluator will use terminology appropriate to evaluation science. Terms like “formative evaluation,” “summative evaluation,” “mixed methods,” and “qualitative versus quantitative evaluation” may not be familiar to technical faculty and hence, have little meaning to members of the project team.   

Another challenge is visualizing how the evaluation will fit into the proposed project. Visualization techniques like logic models have been used to link project activities to outputs to outcomes. Logic models are most often represented as tables but are word-based and thus may not easily convey evaluation processes and procedures to technical faculty. An alternative technique to logic models is the use of “road maps” as a visualization tool. Road maps are similar to flowcharts in that they use blocks to represent essential elements and arrows to indicate important interactions between project components. Technical faculty will be familiar with flowcharts and diagrams, such as circuit schematics that are similar to road maps. 

Evaluation frameworks can also effectively overcome communication/language and visualization barriers. An effective framework shows how project activities and evaluation processes can be used to determine the “merit” or “worth” of the project being proposed. Most individuals will be familiar with the term “value,” but we will differ in how to quantify it. An evaluation framework focused on value creation can be used to visualize and measure the merit or worth of a project.  

A Value-Creation Evaluation Framework 

Evaluation often stops short of answering the “so what?” question. What is the impact of the investment that a grant-funded project represents? How can an evaluation plan focus on this critical question? As an early-career project evaluator and building on years of experience as a STEM technician educator and NSF ATE grantee, the author developed a framework incorporating evaluation methodologies to measure a project’s merit or worth (value). This value-creation evaluation framework was adapted from a conceptual framework developed by Etienne Wenger, Beverly Traynor, and Maarten de Laat for communities of practice. Their value-creation evaluation framework is described in a paper entitled “Promoting and Assessing Value Creation in Communities and Networks: A Conceptual Framework” [2]. Their framework measured value creation in terms of five-cycles: 

1. Immediate Value: Activities and interactions 

2. Potential Value: Knowledge capital 

3. Applied Value: Changes in practice 

4. Realized Value: Performance improvement 

5. Reframing Value: Redefining success 

The following is a brief introduction to Wenger’s five levels of value creation. 

Cycle 1 – Immediate Value – Activities and Interactions: “Activities and interactions can produce value in and of themselves. One can get an answer to a question, a solution to a problem, or help with a challenge [2]. Immediate Value is created or produced during an activity. 

Cycle 2 – Potential Value – Knowledge Capital: “Not all value produced is immediately realized. Activities and interactions can produce ‘knowledge capital’ whose value lies in its potential to be realized later [2].” This can include changes in perspective or attitude. Potential Value is created as a result of an activity. 

Cycle 3 – Applied Value – Change in Practice: “Leveraging capital requires adapting and applying it to a specific situation. Looking at applied value means identifying the ways practice has changed in the process of leveraging knowledge capital [2].” Applied Value is produced after an activity. 

Cycle 4 – Realized Value – Performance Improvement: “New practices or tools are not enough, even when applied. It is therefore important not to simply assume that improved performance is the case when people change their practice, but to reflect on what effects the application of knowledge capital is having on the achievement of what matters to stakeholders [2].” Realized Value is the desired product or outcome of an activity. 

Cycle 5 – Reframing Value – Redefining Success: “The last cycle of value creation is achieved when social learning causes a reconsideration of the learning imperatives and the criteria by which success is defined. This includes reframing strategies, goals, as well as value [2].” This cycle is a critical step for continuous improvement as the members of the project team seek to optimize activities, strategies, and methodologies. Reframing Value results from reflection on an activity and the evaluation data collected during Cycles 1-4. 

Value-Creation Example 

The following example will illustrate the use of the value-creation evaluation framework. Suppose your goal is to take better photos of your grandchildren during summer vacation and to produce a photo book to remember the time together. A first step would be to research available cameras, select the best camera to meet your needs and budget, find the best price at a local camera store or online, and purchase the camera to best meet the stated goal. This creates Immediate Value.   

Having purchased the camera, Potential Value resides in the photographer’s ability to use it to take photos at some future point. That Potential Value may or may not be realized. 

Applied Value is created when the camera is actually used to take summer vacation photos of family members and document the time spent together. This Applied Value is created when the camera is actually used. 

Once a library of photos has been taken, one of a number of software programs can be used to create a photo book documenting the vacation, which can then be printed. This achieves the goal. So, producing the photo book represents Realized Value.   

The final cycle is to use self-analysis of the process and constructive feedback from family members about the photobook to determine what may be done to improve future vacation photobooks. For example, next time more candid photos than posed photos would make the photobook more realistic and meaningful. This will improve the process and creates Reframing Value. 

The photo book example is hypothetical. Numerous examples may be found in everyday life. How many times has a book been purchased but never read, seeds purchased but never planted, or a workshop attended, and the workshop notebook with guides for improvement never again opened? Good intentions without follow-through prevent the achievement of goals. Evaluation provides an accountability tool that can help turn good intentions into reality. 

A Case Study 

To understand how the value-creation evaluation framework works, an evaluation of a series of linked ATE projects will be used to illustrate the application of the Value-creation framework. Over the past decade, this framework has been used to show the merit, worth, and return on investment of Mentor-Connect: Leadership Development and Outreach for ATE, a series of special projects funded by the National Science Foundation’s Advanced Technological Education Program (DUE 1204463, 1501183, 1840856, and 2227301). Referred to as Mentor-Connect, this initiative is responding to a documented need to promote the achievement of the goals of the NSF ATE Program. 

Prior to 2012, NSF’s ATE Program had a preliminary proposal program. Prospective Principal Investigators had the option of submitting a five-page preliminary ATE proposal for review. Program Officers and selected reviewers would read the proposal and offer suggestions for strengthening the proposal prior to the final submission of a full proposal. When the preliminary proposal program was discontinued, most prospective grantees were faced with accessing and interpreting the NSF solicitation and associated grant policies and procedures on their own. Some prospective grantees decided it was much too daunting a task and did not even try.    

By 2012, the ATE Program was 20 years old, but only one-third of the nation’s 1,200 two-year colleges had received grant funding from the ATE Program. Conversely, approximately 800 two-year institutions had not received an ATE award. Furthermore, of the 1,200 two-year institutions, approximately 260 were small rural colleges [3] that often lacked the grant expertise and experience needed to compete in the competitive ATE grant program at NSF. 

Several seasoned ATE grantees who had been providing ad hoc mentoring assistance for grantees in the ATE Program were encouraged by ATE Program Officers to create a mentoring program to assist new-to-ATE institutions with preparing and submitting competitive new-to-ATE proposals. Elaine Craft, based at Florence-Darlington Technical College, SC, led this effort. The first Mentor-Connect award, DUE 1204463, was designed to create a regenerative, sustainable process for peer mentoring for the ATE Program. The program focused on proposal development and leadership capacity-building among those who educate technicians in high-technology fields that drive our nation’s economy. To achieve this goal, the Mentor-Connect Program was designed to [4]: 

• Fill a void created by the elimination of the preliminary ATE proposal review process. 

• Address the fact that roughly two-thirds of community colleges have not received an award from the National Science Foundation (NSF) ATE Program in the last ten years. 

• Better manage the growing number of requests received by ATE Center Principal Investigators and NSF Program Officers related to grant proposal development and project management. 

• Develop grant writing skills among community college faculty who lack grant development staff (or sponsored research officers) at their institution. 

Mentor-Connect project objectives were to: 

• Attract and prepare new and diverse cadres of ATE Principal Investigators (PIs) from community colleges. 

• Transfer knowledge from experienced PIs to less-experienced or new PIs. 

• Prepare experienced ATE PIs to serve as peer mentors. 

Three subsequent grants (DUE 1501183, 1840856, and 2227301) enabled the Mentor-Connect Project to continue cohort mentoring over the past decade. Ten cohorts of two-year colleges have completed the ATE funding cycle, Cohort 11 proposals are currently in the review cycle, and Cohort 12 began proposal development in January 2024.  

Mentor-Connect Roadmap 

The theory of change for the Mentor-Connect Project is very straightforward. From a pool of applicants, a cohort of 20 mentee colleges would be selected. Each mentee team would consist of two faculty members, one being designated as the Principal Investigator (PI) and the other as the Co-Principal Investigator (co-PI). Each college would be assigned a mentor to guide them through the proposal development/submission process. The mentorship program (intervention) includes instruction and technical assistance in addition to mentoring. The intervention begins with an intensive three-day workshop that includes an overview of the ATE program, a description of essential proposal components, a mock panel review of actual ATE grant proposals, and a presentation of elevator speeches that allow each Mentee team to describe their project idea. Some workshop participants liken the workshop experience to “drinking from a fire hose,” and it reflects the steep learning curve that exists when technical faculty begin preparing their first NSF ATE grant proposal.  

As proposal development begins, the mentoring process is supported by webinars that are focused on specific aspects of proposal development, e.g., evaluation, forms, and budget and budget justification. Also supporting the mentoring process are proposal development resources accessible on the Mentor-Connect.org website. Online resource materials are designed to answer frequently-asked questions. Mentors and mentee teams communicate primarily through email messages and telephone/online conference calls. Nine months of mentoring culminates with the submission of proposals in early October. The target metric set for each cohort is an 80% submission rate (16 submissions by 20 mentee teams). Through the first ten cohorts, it was envisioned that nearly all of the proposals would be in the Institutions New-to-ATE proposal track within the ATE Program. 

As Mentor-Connect selected the first cohort of Mentee colleges in 2012, the roadmap shown in Figure 1 was used to visually represent the project. In the left-center of the roadmap are blocks representing “Mentors” and “Community Colleges (Mentee teams),” and the double arrow depicts the interaction (i.e., mentoring) between the mentors and mentees.   Three “value investments” support the mentoring process: Grant-writing Workshops, Webinars, and Website Resources. All three value investments support Mentors and Mentees during the nine-month mentoring process.   

On the right side of the roadmap is a large block labeled “ATE Community.”   The arrow between Community Colleges and the New-to-ATE Projects block depicts the proposal submission process for our Mentees. Successful proposal submission provides entry to the ATE community.   

The arrow at the bottom of the roadmap depicts the regenerative nature of the Mentor-Connect Program. It is envisioned that Mentees who receive a New-to-ATE award will move up to an ATE project award, gain experience with ATE awards over time, and then ultimately become mentors themselves for future cohorts. This is the regenerative, capacity-building nature of Mentor-Connect. 

Evaluating the Mentoring Process 

Immediate and Potential Value 

Evaluating the cohort selection process is mainly quantitative: the number of teams accepted into the cohort from the applicant pool. As shown in Figure 2, the number of applications varied from a low of 23 applicants (Cohort 9) to a high of 37 applicants (Cohort 6). The overall acceptance rate is approximately 70%. Those not accepted into a cohort were offered a one-on-one mentoring session to discuss how their project idea and their Mentor-Connect application might be improved and to explore other programs that may be a better fit for assistance in meeting their goals, e.g., the MentorLinks Project for program improvement. 

Evaluating the two Mentor-Connect in-person workshops relied mainly on post-workshop surveys, and the surveys yielded “Immediate Value”. Over the years, this data has been consistently positive. Workshop participants have been very pleased with the workshop content and rated all aspects of the workshop as “High Value” or “Very High Value.”  However, the data has become less valuable as an evaluation tool over the years since the data have not significantly changed from year to year. Suggestions such as the need for more Mentor-Mentee work time have been addressed while being attentive to the need to balance information transfer with Mentor-Mentee interaction and budget constraints on workshop length.    

One question on the post-workshop survey pointed to “Potential Value”. The last question on the survey asks, “On a 10-point scale with ‘10’ representing ‘very confident,’ how confident are you that you will be able to use the workshop material to begin developing an ATE proposal?”  Leaving the workshop, most participants are very enthusiastic and were very confident that the workshop material, along with mentoring, would result in ATE proposal submission. 

Likewise, post-webinar surveys were administered at the end of each webinar. Over the years, the results of these surveys have shown a high level of satisfaction with the webinars. Most participants view the information presented as vital for preparing their proposal in the days ahead. Quick Reference Guides based on each webinar help participants find needed information without having to listen to the whole webinar a second time. Comments on participant surveys and questions asked during and at the conclusion of webinars guide content improvement to prepare subsequent cohorts of prospective grantees better. 

Applied Value 

The mentoring process is monitored through four progress reports sent out and collected by Emery DeWitt, Mentor-Connect Project Manager. These progress reports identify mentee teams that may require intervention by Mentor-Connect staff to address specific problems or issues as they arise. However, some problematic situations involve internal issues at a college. The project team and mentors do not intervene but strive to equip participants with leadership skills to help them resolve issues they encounter at their college. These data measure “Applied Value.” 

Creating “Applied Value” depends on a cadre of experienced mentors. To date, 34 different individuals have served as mentors. Having a stable group of mentors is critical to generating “Applied Value” in the Mentor-Connect Program. Each August, individual mentor interviews are conducted via phone, or more recently via Zoom, with each mentor currently mentoring in the cohort. During the interview, mentors are asked for suggestions on how to improve the workshops, webinars, and resources to help them better mentor their mentee colleagues. Mentors are also asked to describe their progress in mentoring each of their mentee teams and are asked to predict whether their mentee teams will be able to submit an ATE proposal in October. Cohort mentoring generates “Applied Value.” 

Realized Value 

Following the submission deadline, mentee colleges are asked to voluntarily submit copies of their ATE proposals to Mentor-Connect.  Using these proposals and by searching the NSF awards database, data is collected to document the number of proposals submitted, proposal numbers, amount of the request, and proposal type. The target metric for Mentor-Connect is an 80% submission rate for each cohort. This data measures “Realized Value.” 

Figure 3 shows the number of submissions per cohort. Only Cohort 9 fell below the 80% target, most likely a result of the disruption caused by the COVID-19 pandemic. During the pandemic, faculty were under increased stress as they converted their courses for online delivery while simultaneously working on their ATE proposal. Over ten cohorts, the Mentor-Connect project has achieved an 83% submission rate. 

Reframing Value 

Beginning with Cohort 2, evaluation data prompted the Mentor-Connect Leadership Team to reframe what success looks like. The following changes were made to Mentor-Connect processes and mentor activities. Reframing aims to increase the value generated in different activities in the Mentor-Connect Project. 

Cohort 2: Feedback from mentors indicated that another face-to-face meeting between mentors and mentees during the summer would enhance the proposal development process, provide an opportunity to make changes, and get some mentee teams back on track prior to submission. Hence, a second, one-day, in-person workshop was added, and this workshop is now held each July just prior to the annual High Impact Technology Exchange Conference (HI-TEC). An important outcome of this workshop is the creation of a timeline for the final two months leading up to the proposal submission deadline, typically the first Thursday in October. 

Cohort 6: Initially, the Mentor-Connect Leadership Team selected mentors from among the most experienced grantees in the ATE grantee community. This approach worked to launch the project but was not optimal for expanding participation over time.   To address the growing need for more and better-prepared Mentors, a formal Mentor Fellows Internship Program was implemented in conjunction with Cohort 6. Mentor Fellows were selected through a formal application process. Those selected to be Mentor Fellows were each assigned to shadow and work with an experienced mentor for one year. In addition, Mentor Fellows participate in all Mentor-Connect grantee preparation activities throughout the year to inform them of current NSF ATE guidelines and expectations while enhancing their grantsmanship skills. Based on their mentor’s recommendation and observation of the Mentor Fellow by members of the Leadership Team, it is then determined which Mentor Fellows who successfully completed an internship will be invited to serve as a mentor the following year. The Mentor Fellows Program builds mentoring capacity for the ATE Program and Mentor-Connect Project and thus supports the “regenerative” part of the Mentor-Connect Project. 

Cohort 7: For earlier cohorts, only one hour of mentoring via phone was provided to those mentees whose proposals were “Declined” (not funded). Some ad hoc mentoring was also provided by the mentor who had been working with the mentee team. For Cohort 7, Mentor-Connect expanded mentoring services to those whose proposals were Declined by adding two additional mentoring assistance programs: Second-Chance Mentoring and Moving-Up Mentoring. Second-Chance Mentoring provides colleges that submitted New-to-ATE or ATE Project proposals that were Declined with additional help in responding to reviewer feedback and strengthening their project idea and proposal for re-submission the following October. Moving-Up Mentoring is for those colleges who have received a New-to-ATE award, have completed or are nearing completion of the project, and want to “move up” to an ATE project.  Both of these mentoring initiatives provide a dedicated mentor for one-on-one mentoring and serve to encourage continued participation in the ATE Program 

Cohort 9: The ATE Program began restructuring the funding opportunity for ATE Centers in 2018. With this restructuring came the expectation that ATE Centers would support other ATE grantees with discipline expertise and resources and mentor educators and encourage them to submit ATE grant proposals. Recognizing the primary role of ATE Centers as leaders in specific disciplines of advanced technological education and Mentor-Connect as a leader in specifics of NSF ATE grant proposal preparation, Mentor-Connect has partnered with ATE Centers to provide Co-Mentoring. Participating ATE Center designates persons from their Center to serve as co-mentors. A co-mentor has the option of preparing for this role by completing a Mentor Fellows internship. Co-mentor Fellows prepare to support the mentoring objectives of their ATE Center with no expectation that they will serve as Mentor-Connect Mentors. Co-Mentors support technician educators in the discipline and can rely on Mentor-Connect to support their work with grant proposal development education, allowing prospective grantees to benefit from both of their expertise. Co-mentoring also supports ATE Centers in other ways such as providing educational programs for their participants on the ATE Program, grant funding opportunities, and proposal development strategies.   Co-mentoring increases “Applied Value.” 

Post-Cohort 9: Beginning with Cohort 10, Mentor-Connect broadened the definition of new-to-ATE to expand eligibility for cohort mentoring and build more capacity for seeking NSF grants at two-year colleges. New-to-ATE now includes college teams where faculty members on the team have not previously served as PI or Co-PI on ATE grants. This enables a college that has been a previous ATE grant recipient an opportunity to work with Mentor-Connect to increase ATE funding to the college and expand the number of faculty with grantsmanship knowledge and skills to advance technician education at the institution and beyond. New-to-ATE faculty where the college is not eligible for the Small Project for Institutions New to ATE funding track are guided in preparing competitive proposals for the ATE Projects funding track. 

Post Cohort 10: Beginning with Cohort 11, a pre-mentoring initiative was implemented. Some applications to the Mentor-Connect Program have significant project description weaknesses or are not aligned with the goals of the ATE Program. Rather than rejecting these applications, a limited number of hours of pre-mentoring assistance is offered to these applicants and provides an opportunity for the college to strengthen their project idea or other elements of their application, which will allow them to participate in the Mentor-Connect Cohort and prepare a competitive ATE proposal. If the applicant is not able to address these issues, a referral may be made for the college to apply to another NSF Program. Most recipients of pre-mentoring have been able to adjust project idea, were accepted into the Cohort, and became prospective ATE grantees. Pre-mentoring intervention has increased the number of mentees accepted into a Cohort, and the number of prospective ATE grantees, thereby increasing “Immediate Value.” 

Post Cohort 11: A new initiative called “PI 101″ was implemented. PI 101 is mentoring new PIs to help them implement their newly funded ATE project. This mentoring helps new PIs to become familiar with NSF procedures, e.g., complying with reporting requirements. Instruction and advice on NSF expectations, elements of project implementation, and management of federal grant funds are designed to reduce the frustration that most new PIs experience and will make implementing their project less stressful. By reducing the stress in their first project, it is hoped that PIs will be encouraged to move up to a complete ATE project in the future. Increasing the likelihood that a PI will seek another grant increases “Potential Value.”  

Current Roadmap 

Reframing has enhanced the Mentor-Connect Project to create additional value. Adding Second-Chance Mentoring, Moving-Up Mentoring, Co-Mentoring, and the Mentor Fellows initiative to the Mentor-Connect Roadmap shown in Figure 1 yields an expanded Mentor-Connect roadmap shown in Figure 4. The Second-Chance Mentoring block at the top of the roadmap is for Mentees who have had their first or second ATE proposal Declined and who are revising it for future submission. The Moving-Up Mentoring block at the bottom of the roadmap depicts Mentees who are preparing to move up from a small, New-to-ATE project to a larger ATE project. The Mentor Fellows block in the lower left corner represents the Mentor Fellows Program that prepares members from the ATE community to become future Mentor-Connect Mentors. And finally, Co-Mentors were added to the Mentor block on the left side of the roadmap. The changes to the roadmap are a direct result of the Mentor-Connect Leadership Team using evaluation data over ten cohorts to “Reframe” Mentor-Connect processes and procedures to maximize the value created. 

Evolution of Mentor-Connect 

The table shown in Figure 5 summarizes the evolution of the Mentor-Connect Project from its beginning in 2012 to its current structure in 2024. The table shows the progression in terms of value creation from Immediate Value and Potential Value to Applied Value to Realized Value and Reframing Value. Over the past decade, the Leadership Team has continuously improved Mentor-Connect processes and procedures, resulting in a mentoring program that has produced a significant return on investment to the ATE Program at NSF. 

Mentor- Connect Strategy	Immediate and Potential Value	Applied Value	Realized Value	Reframing  Value
Initial Theory of Change (circa 2013)	Mentee teams In Cohort: new-to-ATE colleges  Workshops/ Webinars  Website  Resources  Mentors	Mentoring  Proposal development  Leadership development	New-to-ATE proposal submitted to NSF  (Metric: 80% submission rate for New-to-ATE proposals)	Add another in-person workshop  Add additional webinars  Expand/update website resources  Implement Mentor Fellows Program  Implement Second-Chance and Moving-Up mentoring  Implement co-mentoring with ATE Centers
Current Theory of Change (circa 2023)	Mentee teams In Cohort: new-to-ATE colleges and new-to-ATE faculty  Workshops/ Webinars  Website Resources  Mentors  Mentor Fellows  Second-Chance and Moving-Up Mentee teams  Co-Mentoring with ATE Centers	Mentoring  Proposal development  Leadership development	New-to-ATE proposal submitted to NSF  ATE project proposals submitted to NSF  (Metric: 80% submission rate for New-to-ATE proposals)  Mentor Fellows become Mentors	Expand Mentee teams from two to four members: Include an administrator and grant professional  Expand the definition of eligible teams to include faculty new-to-ATE  PI 101 Mentoring  Pre-Mentoring applicant intervention  Expand/update website resources

Results and Discussion 

Longer-term Impacts of a first ATE Grant Award 

Evaluation outcomes for Mentee colleges do not stop with the submission of an ATE proposal and subsequent ATE grant award. Seventy-one percent of mentee colleges receive New-to-ATE awards and become members of a very welcoming and supportive ATE community. For many mentee colleges,’ New-to-ATE awards are transformative; the award is the catalyst for changing the grant culture within the institution. The result has been the implementation of procedures and policies that support both pre-award and post-award grant activities for faculty-led grants. The problem-based-learning scenario of developing a project fulfills a vision for preparing and submitting a competitive grant proposal, becoming a grantee, and implementing a project that develops leadership skills among STEM faculty and technician educators who complete this journey, especially if they continue to seek and implement additional ATE grant awards. Leadership skills are evident as individuals lead within their college, across the ATE Community, and beyond, including becoming a Mentor for prospective ATE grantees.  

Funded Proposals 

The success of Mentor-Connect is reflected at interim milestones as new educators are connected to the NSF ATE Program and culminate with the success that a funded proposal represents. Data are available for the first 10 Cohorts. Cohort 11 proposals are being awarded in 2024, and prospective grantees in Cohort 12 are currently developing proposals for submission in October 2024.  

Out of 164 New-to-ATE proposals submitted, 117 were funded, with an overall success rate of 71%. Figure 5 shows the number of proposals submitted and awards received by the cohort. The success rate for New-to-ATE proposals ranged from a low of 47% for Cohort 8 to a high of 88% for Cohort 6. 

Cohort 1-10 New-to-ATE awards total $30,063,197. However, the value to the college goes beyond the dollars received from NSF. This first ATE award can be the catalyst for change within an institution. The award will increase awareness of the availability of federal funding. It can increase visibility for the program and college within the community and foster new and strengthen existing industry and community partnerships. Faculty become involved in a national community of technology educators and establish their networks with colleagues from other institutions. And, the experience of getting their first grant can stimulate submission for future grant proposals using the knowledge gained from their first successful proposal. 

For evaluation purposes, each mentee team that submitted an ATE proposal is asked to voluntarily share their panel reviews with Mentor-Connect. The panel reviews indicate both weaknesses and strengths in the proposal. As an aggregate of proposals, reviewer comments may indicate areas of proposal development that should be highlighted in Mentor-Connect workshops. The reviewer ratings are also analyzed for “spread.”  That is the range between the highest rating for a proposal and the lowest rating it received. One finding was that the spread in ratings increased during the pandemic when NSF changed from in-person panels to “virtual panels.”  This suggests that it is harder for virtual panels to influence and moderate extreme perspectives on the merits of a proposal. 

Site Visits and Outcomes Harvesting 

To determine the impact of receiving their first ATE award, site visits were used to implement what we called “Outcomes Harvesting.”  By visiting each mentee college near the end of their project and interviewing the Principal Investigator, project staff, administrators, and grant personnel, outcomes from the NSF award could be “harvested.”  Approximately 40 mentee colleges in Cohorts 1-4 were visited prior to 2020 when COVID-19 forced travel limitations. For Cohorts 5-7, site visits were conducted virtually using Zoom. 

One of the important findings during an in-person site visit after a first-time grant is being implemented is the degree to which the institution is developing a “culture-of-grants.”  A culture of grants can be defined in many ways. For the purposes of this paper, it is defined as “the set of shared attitudes, values, goals, and practices that support grants;” attitudes that grants are an asset to educational programs and can be a catalyst to improving educational programs and student learning, values that drive faculty and administrators to invest time and effort in seeking educational grants, goals that include educational materials development, laboratory improvement, and faculty enhancement, and, practices that encourage and support faculty as they step beyond their role as a classroom teacher. 

Site visits have indicated a range of cultures related to grants. Some two-year colleges have no grant professionals on staff and do little to encourage faculty to seek grants. Other institutions have embraced grants as enablers for program improvement and faculty development. Some institutions now include faculty in grant planning and advisory committees, whereas before becoming ATE grantees, grant seeking was predominantly administrator-driven. 

However, the best indicator of the development of a culture of grants is to track the grant activity following an institution’s initial ATE award. Evaluation of this metric requires periodic “mining” or “searching” the NSF awards website by Mentee institution. Has the institution received a subsequent NSF ATE award, that is, have they “moved up” from a New-to-ATE project to an ATE project, consortium, or center award? Has the institution also received an award from another NSF program, e.g., an S-STEM or Hispanic-Serving Institution Program grant? 

Mining the NSF awards website in November 2023 for Cohorts 1-10 awards identified that in addition to the 117 Mentor-Connect mentee colleges that have received New-to-ATE awards on their first try, totaling $30,063,197, an additional 25 mentee colleges received 25 second-chance, New-to-ATE awards totaling $8,010,850. Of these 142 colleges that received New-to-ATE awards, 36 have moved up to larger ATE awards totaling $34,536,793. These colleges also applied for and received 20 S-STEM awards totaling $22,247,095 and 32 other NSF program awards totaling $24,904,967. Since participating in the Mentor-Connect Program, 142 mentee colleges have received approximately $120M in grants from NSF programs. 

Faculty Leadership Development 

Community college culture sees the role of faculty as classroom teachers. Activities outside the classroom are often viewed as unnecessary and thus are rarely recognized or supported. Teaching loads are heavy, ranging from 18 contact hours/week to as many as 24 contact hours/week. With the associated office hours, preparation time, and assessment of student learning, most faculty view their jobs as more than full-time. So, when a faculty member is asked to participate in grant-related activities, their response may be less than enthusiastic. Even if provided with some “release time” for completing grant work, faculty too often lack a vision for grants, are unable to see the potential benefits, and do not pursue the opportunity. 

Fortunately, there are faculty members who do see the advantages of grants. The motivation is usually a desire to improve student learning and to provide real-world experiences in the instructional program. For those faculty, the return on investment can be significant. For example, attending the Annual ATE Program Principal Investigators’ Conference can be an eye-opening experience for faculty that expands their vision for technician education and builds their professional network. 

Faculty leadership development through the Mentor-Connect Program is “leadership development by doing.”  Getting a proposal through an institution’s internal approval processes and implementing a funded project requires faculty members to work outside of their comfort zone as they interact with other departments within in their institution, faculty at different colleges, and Program Officers at NSF. 

Examples of faculty leadership development include faculty at Northland Community and Technical College (Unmanned Autonomous Vehicles), Sommerset Community College (Additive Manufacturing), Central Oregon Community College (Electric and Hybrid Vehicles), Snow Community College (Agriculture), and Northwest State Community College (Scaling Online and Hybrid Instruction). These are just a few examples of ATE grants serving as catalysts for faculty leadership on a state, regional, or national level. 

A visible measure of faculty leadership development in Mentor-Connect mentees is demonstrated at the ATE Connect sessions at the ATE Principal Investigators Conference. In 2013, 10 Mentor-Connect mentee colleges participated. In 2023, the number of participating Mentor-Connect mentee colleges had grown to 55 colleges. There were a plethora of Mentor-Connect buttons worn by conference participants who are part of the Mentor-Connect family and Mentor-Connect stickers on posters at the two ATE Connects sessions. In addition, mentees also participated in conference sessions and breakfast roundtables and served on conference planning committees. Numerous program sessions are provided at the annual conference each year by new and repeat ATE grantees who first engaged with the ATE Program as Mentor-Connect Mentees. 

Conclusion 

Coupling a straightforward theory of change with a descriptive visual tool and a value-creation evaluation framework provides a robust way to monitor and evaluate an ATE project as well as communicate an effective evaluation plan to project team members and stakeholders. The theory of change describes how the project will get from inputs and activities to outputs, outcomes, and finally to project goals. A visual tool such as a roadmap presents the theory of change in a graphical format that can be easily understood and remembered. A value-creation framework defines types of “value” that will be created, measured, and analyzed so that the project team can make improvements to the project. It helps answer the “so what?” question that is of utmost importance to funding agencies. 

The Mentor-Connect example illustrates how a well-defined theory of change coupled with a roadmap and value-creation evaluation framework can be used to determine the merit or worth of an ATE project. The goal of the Mentor-Connect Project is to help two-year colleges that have not received awards from the NSF ATE Program to submit competitive ATE proposals. Over 10 cohorts of mentee colleges and 208 mentee teams have received mentoring, with 83% submitting New-to-ATE proposals. Of the submitted proposals, 117 colleges (71%) received awards. 

The impact of the Mentor-Connect Program goes beyond getting an ATE award. “Outcomes Harvesting” based on in-person and virtual site visits has uncovered outcomes beyond receiving their first ATE award. Outcomes include the development of a culture of grants to support future grant activity and faculty becoming leaders in technician education as a result of being a part of the larger ATE community. Colleges have become more adept at seeking, receiving, and managing grant funds. Partnerships have increased and become more meaningful, and colleges have garnered recognition and respect as NSF grantees. Mentees have moved up to larger ATE projects and even consortium and center grants. Regeneration and capacity-building among two-year colleges to engage with the NSF ATE Program is being realized as past mentees now serve as mentors for prospective grantees in each subsequent cohort of colleges. 

Acknowledgments. This work was supported by the National Science Foundation (NSF) under awards 1204463, 1501183, 1800678, and 2227301. 

Disclosures. The authors declare so conflicts of interest.