Systems Testing Excellence Program (STEP)

UofM’s STEM is the largest academic group devoted to software testing in the U.S.

The Systems Testing Excellence Program (STEP), a unit of the University of Memphis’ FedEx Institute of Technology (FIT), is the largest academic group devoted to software testing in the U.S. STEP includes professors from several UofM departments and has affiliate faculty at the University of Oklahoma, Brigham Young University, Southern Methodist University, and others. STEP was founded in 2006 as a collaboration between the UofM and the FedEx Corporation and since that time has provided training in software testing to hundreds of software professionals from all the major corporate and government entities in the Mid-South region. STEP has also had training contracts with the U.S. Defense Department’s (DOD) Joint Interoperability Test Command (JITC), conducting training programs at Ft. Huachuca, AZ, Ft. Meade, MD, and at the Indian Head Naval Facility, MD.

STEP is currently performing a $1.6 million, five-year software test and evaluation research and training contract from the U.S. Department of Homeland Security (DHS). The training program involves one-week classes for DHS headquarters personnel from all the DHS components. These include the U.S. Coast Guard, Transportation Security Administration (TSA), Secret Service, Customs and Border Protection (CBP), Federal Emergency Management Agency (FEMA), and others. These classes have been held in Washington, D.C., as well as virtually. For this calendar year, DHS has granted STEP two $95,000 research grants involving the state-of-the-art topic of the use of artificial intelligence in software testing. These are being conducted by UofM professors Dr. Mark Gillenson, Dr. Pavan Mulgund, and Dr. Ankur Arora, and graduate students from the Fogelman College of Business and Economics’ Department of Management Information Systems, with the assistance of a STEP affiliate professor from the University of Oklahoma.

One of the research projects is entitled “The Use of Large Language Models for Test Case Generation.” Test case generation is a key process in software testing and there have been several methods for generating test cases. Recently, Large Language Models (LLM) such as ChatGPT and others have emerged. This transformative technology is capable of responding to submitted prompts and producing text and other artifacts in response. The research project that STEP is undertaking seeks to develop a generalized process for creating LLM prompts for generating software test cases. We intend to develop methods of instructing LLMs about the comparative risk of different features of applications, which will guide them in generating the most useful test cases in an environment of limited testing resources. Finally, we intend to compare the test case output of LLMs with test cases produced by traditional manual and algorithmic methods based on efficiency, adaptability, domain specificity, complexity and interpretability using experimental approaches. 

The other research project is entitled, “Developing Effective Test Cases for Assessing Ethical Principles in AI Systems – A Design Science Research Approach.” This project aims to solve the problem of the lack of standardized and effective test cases for evaluating the ethical principles in AI systems. It will identify key ethical principles that need to be assessed in AI systems, develop a framework for creating test cases that can effectively evaluate these ethical principles, and validate the effectiveness of these test cases through empirical research. The research design will include a mixed-methods approach combining a design research approach (for framework development) and qualitative research (case study) for validation. We expect the results to include a comprehensive set of test cases for assessing ethical principles in AI systems, a validation framework demonstrating the effectiveness of these test cases, and recommendations for AI developers on incorporating ethical considerations into AI system development.

For more information on this program, contact Dr. Mark Gillenson, Director, at mgillnsn@memphis.edu.