Securing the Future: ZotGPT and Cyber@UCI Unite to Fortify Generative AI Defenses
Cyber@UCI, a student-led cybersecurity club
March 19, 2025
As AI continues to reshape our digital landscape, UC Irvine is taking bold steps to ensure the security of ZotGPT Chat, our signature generative AI solution. A novel collaboration between ZotGPT and Cyber@UCI, a student-led cybersecurity club, is setting new standards for AI security testing and student engagement in real-world cybersecurity challenges.
A Unique Partnership
UC Irvine's artificial intelligence team quickly realized they required additional resources to further secure ZotGPT. Unlike large corporations that have budgets to routinely employ red team consultants, UC Irvine currently depends on automated security testing tools and does not have dedicated full-time employees to perform targeted security testing.
Jeff Picco, Security Engineer and a member of the AI Work Group, formulated a plan that would recruit the assistance of Cyber@UCI. He envisioned a mutually-beneficial partnership that would give the students real-world experiences. For Picco, this effort with ZotGPT could signify the beginning of a greater endeavor that would recruit the services of Cyber@UCI to test other campus services. To that end, Picco connected the club to Chris Price, AI Architect and Lead AI Developer.
"It's important to highlight the efforts of OIT Information Security and Cyber@UCI in helping us secure ZotGPT," said Price. "We hope to set the stage for similar collaborations here at UC Irvine and encourage other UCs to evaluate the same."
The collaboration grants students access to staging environments for pre-release code testing, a rare opportunity for undergraduate students, especially in a cutting-edge field like generative AI. This opportunity will prove to be valuable as the students involved progress in their field and seek internship and employment opportunities.
Meet the Cyber Sleuths
Leading the charge in this initiative is Drew Levy, a junior transfer student with a passion for ethical hacking and offensive security. Levy's journey into cybersecurity began with security competitions and an ethical hacking class at his community college. Now, he's at the forefront of testing UCI's AI systems, focusing on web interfaces and attempting to "jailbreak" language learning models (LLMs).
"AI security testing is especially challenging due to the technology's novelty and the lack of well-defined testing methods. With AI chatbots only recently gaining prominence, ethical hacking approaches for AI have not been fully developed, with most learning resources not addressing such topics." Levy noted.
Despite these challenges, Levy and his team have made significant headway in identifying potential vulnerabilities.
Unveiling the Vulnerabilities: Key Findings in AI Security
The student team's efforts have already yielded important findings that will soon be formalized as a penetration test report. This report will not only serve as a crucial document for UCI's cybersecurity efforts but also as a testament to the students' growing expertise in the field.
The testing methods employed by the Cyber@UCI team are comprehensive. They include:
- Testing input boundaries by entering unexpected values
- Attempting to extract system prompts from chatbots
- Exploring web application security while researching AI-specific vulnerabilities
- Testing unique ID vulnerabilities and character limit constraints
- System Prompt Injection to receive sensitive information from chatting with the AI
"We're not just looking at traditional web security," Levy emphasized. "We're exploring the unique vulnerabilities posed by AI systems, such as the ability to extract sensitive information purely through prompts - a skill that requires little technical expertise to pull off."
Beyond the Code: The Impact of the Collaboration
The partnership between ZotGPT and Cyber@UCI is proving to become a transformative educational experience as students gain hands-on experience in penetration testing.
Steven Ngo, a Ph.D. student involved in managing the collaboration and President of Cyber@UCI, noted the delicate balance they're striking.
He said "We are used to learning through completing cybersecurity challenges that are designed to have vulnerabilities and have a 'flag' that always exists for us to find - when trying to test real-world systems, there is no guarantee that there is a vulnerability. The real value in this opportunity is giving students a unique opportunity to learn how to work within real-world systems while still learning without having that on-the-job pressure to find vulnerabilities."
Among the project supporters is Professor Josh Garcia, Ngo's Ph.D advisor. Garcia is acknowledging the significant educational value of this initiative by supporting Levy and the team with independent study units. He also advises Ngo on how to best support and guide Levy and his team.
Garcia said: "I'm excited to support the Cyber@UCI students and OIT in helping to provide security and privacy assurance for OIT's generative AI solutions. It's always an honor working with smart and passionate students like Steven [Ngo] and Drew [Levy] on this endeavor. Beyond simply applying existing approaches to hacking, breaking, or assessing gen-AI solutions like ZotGPT, this endeavor further allows the team to identify deficiencies in existing approaches to analyzing generative AI solutions, opening further research opportunities to expand the capabilities of the testing and analysis of such solutions."
The Road Ahead
As AI continues to evolve and integrate into our daily lives, the collaboration between the UCI ZotGPT team and Cyber@UCI is an example of proactive security measures and innovative education.
"What makes this partnership truly special is the dual benefit it provides. On one hand, we're strengthening our AI systems against potential threats, which is crucial as we continue to expand our use of this technology across campus," said Tom Andriola, Chief Digital Officer at UC Irvine. "On the other hand, we're offering our students invaluable, hands-on experience in one of the most critical and rapidly evolving fields in tech. By involving students in securing our AI systems, we're not only enhancing UCI's technological infrastructure but also cultivating the next generation of cybersecurity leaders."
This partnership serves as a model for how universities can address emerging technological challenges while providing unparalleled learning opportunities for their students. As we look to the future, it's clear that UCI is not just keeping pace with the AI revolution – it's helping to secure it.
Students interested in joining Cyber@UCI can join the group's Discord.