Analysis of the Impact of Train-the-trainer Workshops on Robotics Education

Introduction

In 2020, and for the first time in history, the electronics industry became the largest market for industrial robots ​[1]​, taking over that position from the automotive industry ​[2]​,​ [3]​, which dominated the market throughout modern history. The most recent report by the International Federation of Robotics shows that the demand for industrial robotics grew rapidly, vertically and horizontally, and has become a major element in multiple industries (Table 1 below). Industries that were almost absent in the last 20 years have become major markets for industrial robots, such as the metal and machinery, chemicals, and food industries.

Table 1. Market share of industrial robotics by industry 

Industry	2022 ​[4]​	2016 ​[2]​​ [3]​
Electronics / Electrical	28.4 %	36.9 %
Automotive	24.6 %	41.7 %
Metal and Machinery	11.9 %	11.6 %
Plastics and Chemicals	4.3 %	6.5 %
Food	2.7 %	3.3 %
Other industries	28 %	~ 0 %

Intelligent industrial robotics, also known as collaborative robots, constitute one of the technologies that emerged as a defining feature of Industry 4.0 ​[5]​. They are industrial robots that are equipped with sensory and perception systems that allow them to work collaboratively with human operators without the need for safeguarding. Despite the great potential for collaborative robots to improve production by as much as one million folds ​[6]​,​ [7]​, they have been shy in penetrating the industrial robotics market due to hesitation by industries to deploy the robots without safeguarding. Consequently, they only occupied 10% of the industrial robot market share in 2022 ​[8]​.  

There are over 100 definitions of “Industry 4.0” that were reported in the literature  ​[9]​, but they share the same set of enabling technologies that include, among others, advanced robotics, the Internet of Things (IoT), and 3D printing. A term that can be used interchangeably with Industry 4.0 is “Smart Manufacturing,” which was more commonly used in the United States and was officially introduced in U.S. Congress in Congressional Bill S.1054. The National Institute of Standards and Technology (NIST) defines Smart Manufacturing Systems as systems that are “fully-integrated, collaborative manufacturing systems that respond in real time to meet changing demands and conditions in the factory, in the supply network, and in customer needs.” One of the keywords in the NIST definition is “collaborative,” which describes human-machine collaboration, one of the major applications of incorporating Artificial Intelligence (AI) in the industry ​[10]​. 

One of the factors that contributed to the rise in the demand for industrial robotics by industries other than the manufacturing industry is the outbreak of the Covid-19 Pandemic, which resulted in a sharp increase in the need for industrial robots for no-contact material handling in applications beyond the manufacturing industry ​[11]​. The main operational advantage of collaborative robots over traditional robots is the significant flexibility that can potentially be achieved through the less restrictive standard safety requirements. For example, for operating traditional robots, safety standard ISO 10218-2 requires robot safeguarding using physical barriers or electro-sensitive protective equipment or other means that are compliant with the standards IEC 61496-1 and IEC 62046 ​[12]​,​ [13]​,​ [14]​. While for collaborative robots, the principal standard is the ISO/TS 15066 standard, which guarantees operator safety by limiting robot speed and distance and restricting the momentum of the robot such that any collision with an operator will not result in an injury without the need for any physical safeguarding ​[8]​.  

Workshops and seminars are commonly used effective tools for faculty development and professional development ​[15]​. One of the objectives of the workshops discussed here aims to answer the need for a trained workforce that can handle collaborative robots in advanced manufacturing. The train-the-trainer workshops that were developed included speakers from industry, industry tours, and hands-on training on collaborative robots and robot vision systems. The workshops were funded by the National Science Foundation’s Advanced Technological Education program.  

Methods  

Six workshops were organized in the summer of 2021, the summer of 2022, and the summer of 2023. The target population for the workshops was educators of Science, Technology, Engineering, and Mathematics (STEM) in community and technical colleges and high schools, primarily in the states of Tennessee and Alabama, and neighboring states. Three workshops were offered in Chattanooga, TN, two in Bessemer, AL, and one in Smyrna, TN. The research was designated as exempt from IRB oversight by the Institutional Review Board (IRB) of the University of Tennessee at Chattanooga.  

Each of the workshops was two days long and included speakers, factory tours, and hands-on training on intelligent industrial robotics. All training was conducted on-ground, a FANUC CR-7iA/L collaborative robot unit was the primary training unit that was used for the workshops at Chattanooga State and Motlow State Community Colleges, and a Sawyer collaborative robot by Rethink Robotics was the primary training unit used for the workshop at Lawson State Community College in Bessemer, AL. Other industrial robot units that were used or demonstrated during the training include Motoman HP3JC, ABB IRB 140, KUKA KR5sixx R650, and Fanuc S-430i. Pictures from the workshops are shown in Fig. 1. 

Fig. 1. Photos from the Intelligent Industrial Robotics workshops that took place in the summers of 2021, 2022, and 2023 in Bessemer, AL (top), Chattanooga, TN (bottom left), and Smyrna, TN (bottom right).

To align with best practices, three methods were used to evaluate the effectiveness of the workshops: 

1. Participants are directly assessed by pre-workshop and post-workshop assessment exams, which include assessments of technical knowledge and terms in the fields of industrial robotics and vision systems. 
2. Indirect assessment through post-workshop evaluation surveys that included multiple-choice questions and open-ended questions. 
3. Six-month Follow-up Survey to asses the impact of the workshops on students. 

All evaluations were administered by the independent evaluator. To comply with federal requirements on research ethics, participants had the option to decline to answer any question on the surveys anytime ​[16]​.   

To recruit participants, the project team advertised the workshops to local and regional secondary and post-secondary institutions nationwide through ATECentral, the project website, emails, and other means.  

Results and Discussion  

Effectiveness of Recruitment and Outreach Efforts: 

To measure the effectiveness of the outreach and recruitment efforts, Table 2 shows the location, state, educational level, and sex of applicants who applied to the workshop. As the table shows, a total of 93 applications were received from 13 states to participate in the workshop. A very large percentage of applicants indicated that they hold a PhD or a Masters degree as their highest educational level (76.3 % of total applicants). A small number of applicants indicated that they hold a high school degree as their highest educational level (6.5 % of total applicants). Females constituted 36.6 % of applicants. The results show that the recruitment efforts were successful in outreach to different groups.  

Table 2. State of Residence, Sex, and Highest Educational Level of Applicants. 

Demographic:	Number of applicants
Applicants by State	Applicants came from 13 states: Alabama, Florida, Illinois,  Indiana, Louisiana, New Jersey, New York, Ohio, Pennsylvania, Tennessee, Texas, Washington, West Virginia
Applicants by Sex:
Female:	34
Male:	58
No answer:	1
Applicants by highest degree achieved:
PhD or Doctoral	26
Masters +	45
Bachelors Degree	8
Associate Degree	8
High School Degree	6
Total:	93

Of the 93 applications that were received, 47 applicants participated and completed the workshops. The demographic distribution of the workshop participants is shown in Table 3 below. 

Table 3. Demographic Data for Workshop Participants. 

Demographic:	Number of applicants
Participants by State	Participants came from six states: Alabama, Florida, Illinois,  Indiana, Louisiana, and Tennessee
Participants by Sex:
Female:	24 (51 %)
Male:	23 (49 %)
No answer:	1  (2 %)
Participants by Home Institution Type:
Four-Year Institution	2 (9 %)
Two-year Post-Secondary Institution	20 (43 %)
High School	23 ( 45 %)
Participants by Race:
American Indian	1 (2 %)
Asian	0
Black or African American	21 (45 %)
Hispanic	1 ( 2 %)
White	22 ( 47 %)
No answer given	2 ( 4 %))
Total participants:	47 (100%)

Table 3 shows that the number of participants from underrepresented groups in engineering (American Indians, Blacks, Hispanics, and Women) was considerably higher in the workshops than the national average for the engineering workforce.  

The presence of these under-represented groups in engineering fields continues to be a major concern in the United States. According to the American Society for Engineering Education (ASEE), in 2019 women constituted only 14% of the engineering workforce in the U.S.  Racial and ethnic under-represented groups (Native Americans/Native Alaskans, Hawaiian/Pacific Islanders, Blacks, and Hispanics) constituted only 13% of the engineering workforce ​[17]​. In the workshops of this project, women constituted 51% of the participants. Furthermore, 45% of participants identified as Black. In 2022, a question on the veteran status was added to the workshop applications, for which four participants (8.5 %) identified as veterans.  

It was shown in a previous publication ​[7]​ that the four under-represented groups identified above constitute a smaller percentage of the workforce in engineering compared to their population ratios, as can be seen in Table 4, which shows data taken from the National Center for Science and Engineering Statistics (NCSES) for the year 2019. 

Table 4. Under-represented Groups in  Engineering in the United States, data for 2019 ​[7]​​ [18]​. 

Demographic Group	Percent of Engineering Workforce	Percent of U.S. Population
American Indian/ Native Alaskan, other races, and individuals with more than one race	< 1.60 %	1~ 2 %
Black or African American	5.4 %	12.5 %
Hispanic or Latino	7.9 %	18.5 %
Female	13.8 %	50.8 %

* Numerical values were taken from Tables 1-2, 9-2, and 9-3 of the NCSES report for 2019 and then converted to percentages.  

Effectiveness of the Workshop Activities: 

In total 44 out of the 47 participants responded to the pre-workshop technical assessment exam, and 41 responded to the post-workshop technical assessment exam and survey. Results of the pre-and post-intervention assessments and the post-intervention survey are shown in Table 5. The results show that the average score of participants significantly improved after completing the workshops with an average score of 84 % compared to 23.1 % before going through the workshop.  

Table 5. Results of Measurement Tools Used for Assessment of Workshop Effectiveness 

Measurement Tool/ Metric	Total Responses	Average Score
Pre-Workshop Technical Assessment Exam	44	23.1 %
Post-Workshop Technical Assessment Exam	41	84.0  %
Post-Workshop Evaluation Survey Multiple Choice Questions:	Total Responses:	Strongly Agree or Agree:
My skills/knowledge increased as a result of participating in this workshop.	29	27
Workshop activities were appropriate and reasonable in the time allowed.	29	26

Open-Ended Questions: The surveys included open-ended questions to accurately capture participant opinions about the learning achieved from the workshops. Some of the comments received for the open-ended question: “What is your major takeaway from this workshop?” include:  

• “More like this one to allow additional time to grasp skills.” 
• “The materials presented were broken down to assure understanding by all levels of knowledge of workshop attendees.” 

Comments received for the question: “Would you recommend this workshop to your colleagues? Why or why not?” include:

• “Yes, I will be able to integrate the materials in my classroom, and I gained community resources.”
• “It opened my eyes to real-world experiences my students will face. It was a challenge to do and learn.”

From the assessment tools and the post-intervention survey results shown above, and from the demographic data of participants shown in this section, we can conclude that the workshops achieved their objectives of introducing intelligent industrial robotics to STEM educators particularly from under-represented groups.  

Assessment of the Impact of the Workshops on Students Using Statistical Tools: 

A Follow-up survey was sent to the participants about six months to one year after completing the workshops to assess the long-term impact of the workshops. The term “impact” here is defined as the effect of the training that the faculty member received as a result of this project on their students. Therefore, only the students who were directly impacted by the trained faculty member in his/her classroom were of interest in the follow-up survey. 

Five of the 47 participants responded to the follow-up surveys ( 10.6 %). The low response rate is attributed to several reasons, among which is that some faculty members may have changed jobs during the period between the end of training and the time the survey was sent. In addition to that, the surveys were sent by the independent project evaluator,  with whom the participants only had electronic contact. Furthermore, research studies show that the recent trends in increased overloads of emails received per day have caused individuals to respond to a smaller fraction of emails per day with smaller responses ​[19]​, and response rates to surveys have generally been on the decline in the last few years and are observed across almost all disciplines ​[20]​. Therefore, it was important for the follow-up survey to be short and consume the least amount of time possible to complete it. The survey’s primary focus was to quantify the number of impacted students, as shown in reference ​[21]​. A question was added to the survey in 2023 that asks for the classification of the institution to identify minority-serving institutions. Three of the five survey respondents indicated that their institutions are listed as Predominantly Black Institutions (PBI), which are defined as institutions with more than 40 % of their student body identifying as Black. One respondent indicated that their institution is a Historic Black College (HBCU). Four respondents indicated that they implemented what they learned from the workshops in their classrooms.  

The total number of students impacted six months to one-year after the workshop was completed, as indicated by the survey respondents, was at least 379 students (one respondent indicated that over 25 students were impacted, another indicated that 300 were impacted, and the remaining three responses were: 54,  “none”, and “several”). Therefore, for the five respondents, and taking “several” to be equated to three, on average about 76.4 students with a standard deviation of 126.9 were impacted by each trained instructor six months to one year after completing the workshop.  

To determine the total number of students impacted at a 90% confidence level, we apply the rules of mathematical statistics for a population size of 47 and a sample size of 5 respondents, which shows that the margin of error is 36%. Therefore, it can be stated with a 90% confidence level and 36 % margin of error that 80 % of the instructors that went through the training used the training in their classrooms.  

Furthermore, it can be stated that the number of students per instructor impacted by the workshops is 76 students per instructor with a 90% confidence level and a margin of error of ±36%  or that 95 students per instructor were impacted by the workshops but at a ±40%  margin of error (this is due to the reduction in the sample size since four of the five respondents indicated that students were impacted). Consequently, using the average number spread over the larger sample approach, we can estimate with a confidence level of 90% that 3,572 students ± 1,286 were impacted by the workshops six months to one year after the completion of the workshop. The results of the statistical assessment are shown in Table 6. 

Table 6. Impact of the Workshops on Students 

Statistical Quantity
Average number of students impacted per instructor	76.4
Standard Deviation	126.9
Total number of students impacted	3,572
Margin of Error	±36%
Confidence Level	90%

Challenges and Lessons Learned 

Some of the challenges that were faced in this project include the high differences in levels of educators that participated in the workshop, the limit on the level of accuracy in collecting demographic data due to change in entries by the participants, the low response ratio for the six month follow up survey, and a limit on equitable access to training resources for some rare cases. For the first challenge, the trainers used hands-on exercises with different levels that were given simultaneously to participants. The second challenge arises because demographic data is collected when individuals apply to the workshop, and when they come to participate in the workshops. Some participants choose to answer on one of the two occasions and decline to answer (or change the answer) on the second data collection occasion. This discrepancy could be reduced by only collecting demographic data once in the application form for the workshops. One way to increase the response rate to the six-month follow-up surveys is to give the responsibility of sending participants survey links to the workshop trainers rather than to the project evaluator. Another way of increasing the response rate is to reduce the number of questions to one or two questions only or to provide an incentive for completing the survey. 

Collaboration with other Advanced Technological Education Communities: 

One unique aspect of this project was the collaboration with other centers and projects within the ATE community. The lead student on this work received training on technical paper writing by the NSF- Micro and Nano Technology Education Center (MNT-EC). The project evaluator and a Principal Investigator participated in the DIE in ATE program, a reflective action project created by the NSF- EvaluATE Center. The collaboration with the EvaluATE center resulted in an action plan that included, among others, conducting Diversity, Equity, and Inclusion (DEI) professional development training for the PI team and methods to capture accommodation needs for participants with special needs such as hearing or mobile impairment. For example, in some workshops, the hands-on training and lectures took place in rooms with high ceilings, which caused a hearing difficulty for participants with hearing impairment. To account for this challenge, questions on accommodation needs were added to the workshop application forms to help prepare proper and equitable access for all participants.  

Conclusion  

An assessment of the effectiveness of six train-the-trainer workshops on intelligent industrial robotics is presented here. The workshops included three parts: factory tours, industry speakers, and hands-on training on collaborative robots and robotic vision systems.   The effectiveness of the workshops was measured using pre- and post-workshop technical assessment exams and a post-workshop survey that included multiple-choice and open-ended questions. In addition to that, the impact of the workshops on students was measured using a follow-up survey that was sent to participants six months to one year after completion of the workshops. A total of 93 persons applied to the workshops from over 13 states, of whom 47 participated and completed the on-ground workshops from six different states in the U.S.  Voluntary Pre- and Post- workshop assessments and evaluations were completed to assess the effectiveness of the workshops. Results show that the participants’ knowledge of industrial robotics significantly improved after the workshops, and the vast majority indicated that the training will be used in their home institutions. The estimated total number of students that were impacted by the workshops was 3,572 students ± 1,286 students with 90% confidence 

Acknowledgments. This work was supported by the National Science Foundation- Advanced  

Technological Education program under award numbers 2000651, 2000776, 2000685, 2000682, 2325500, and 1841783 

Disclosures. The authors declare no conflicts of interest.