2026 LLM Software Development Winter School

Abstract

Large language models are increasingly becoming integral components of software systems, yet their evaluation still largely relies on predefined benchmarks and collections of manually constructed inputs. Software testing poses a different question: rather than measuring performance on inputs we already have, how can we systematically explore the enormous space of possible interactions to uncover failures we do not yet know about? In this talk, I will discuss recent approaches for systematically testing LLM-based software systems. I will show how techniques from automated and search-based software testing can be used to explore large natural-language input spaces, guide test generation toward failures, and diversify generated tests to expose different system behaviors. I will exemplify these challenges using conversational systems, where failures may emerge only through sequences of interactions. Drawing on lessons from testing other AI-enabled systems, I will conclude with open challenges around test adequacy, failure oracles, and reproducibility, and discuss how these testing principles could extend to agentic systems.

Bio

Andrea Stocco is an Assistant Professor at the Technical University of Munich (TUM) and Head of the Automated Software Testing field of competence at fortiss. His research focuses on automated software testing, with particular emphasis on testing AI-enabled systems, including deep learning models, autonomous driving systems, and LLM-based software. His work investigates techniques for systematic test generation, search-based testing, and the validation of AI systems across simulation and real-world environments. His research has been published at major software engineering venues including ICSE, FSE, ASE, and in journals such as IEEE TSE and ACM TOSEM.

Abstract

Large language models (LLMs) promise software from a single sentence, but that fluency solves generation, not the harder problem of knowing what to build and when to trust the result. This talk reframes requirements engineering as a second-order control problem: LLMs supply cognitive “energy,” yet compound uncertainty across humans, AI, tools, and organizations. I will present design principles that make unknowns first-class, as well as the conceptualization of requirements space, cognitive-state maps, fog ledgers, and space-aware AI for requirements engineering. Early prototypes will also be presented, showing how human and AI collaboratively drive the exploration, convergence, and validation of requirements. The aim is calibratable trust, not just faster generation.

Bio

Dr. Zhenchang Xing is a Senior Principal Research Scientist and the Science Leader of the Software Systems Group at CSIRO. Dr. Xing’s current research focuses on the interdisciplinary areas of software engineering, human-computer interaction, and responsible AI. He has over 200 peer-reviewed publications in prestigious journals and conferences and has received multiple distinguished paper awards from top-tier software engineering conferences, including the ACM SIGSOFT Distinguished Paper Award, IEEE TCSE Distinguished Paper Award, and ACM SIGSOFT Most Influential Paper Award. Dr. Xing was the Program Co-Chair of ICSME 2020 and will serve as the General Co-Chair of APSEC 2027 and the Program Co-Chair of FSE 2028.

Abstract

AI can increasingly generate and modify software, but generating an interface does not tell us whether that interface actually works well for its users. Many critical software qualities, such as accessibility, design consistency, and freedom from deceptive interaction, are experienced at the user-facing layer and remain difficult for machines to reason about. This talk presents a research agenda for turning such human-experienced quality concerns into machine-assessable software properties. I will show how interface understanding and domain knowledge can enable machines to observe, assess, and improve user-facing software, drawing on examples in accessibility analysis, design repair, and dark-pattern auditing. I will then discuss how this perspective changes as interfaces become dynamically generated and personalized by AI. The broader goal is to make user-facing software quality continuously observable, assessable, and actionable in increasingly adaptive software systems.

Bio

Dr. Jieshan Chen is a Senior Research Scientist and the UI Intelligence research lead at CSIRO. Her research lies at the intersection of software engineering, human-computer interaction, and responsible AI, with a particular focus on software usability and accessibility, UI engineering, deceptive pattern detection, and agentic AI systems. She has published extensively in leading software engineering and HCI venues, and received the ACM SIGSOFT Distinguished Paper Award at ICSE. Her research has also been translated into real-world impact, with research outcomes adopted by major technology companies including Apple and Alibaba. Dr. Chen has organised multiple international workshops, such as RAIE@ICSE 2026 and Harness4GenUI@ASE 2026, and regularly serves on the program committees and as a reviewer for leading software engineering and HCI venues.

Abstract

Manual debugging and repair of test failures during continuous integration checks is time-consuming and disruptive to developer flow. While LLMs have advanced automated program repair, existing offline post-submit workflows fail to provide low-latency agentic assistance before developers context switch. In this talk, we introduce FlowAgent, an autonomous AI coding agent deployed across Google to repair pre-submit test failures in continuous integration. We examine FlowAgent's ReAct generate-and-validate architecture and its pre/post-execution abstention filters designed to meet strict industrial latency and trust constraints. We share empirical findings from its Google-wide deployment (including over 26,000 applied fixes) and discuss the open challenges of integrating autonomous coding agents into real-world developer workflows.

Bio

Spencer Greene is a Software Engineer at Google working on AI-powered developer tooling. His work focuses on designing, building, and deploying autonomous AI agents that assist engineers with code review, test debugging, and automated program repair across Google's monorepo. He co-authored recent work on low-latency agentic program repair presented at the ASE 2026 Industry Showcase. For more see https://spencergreene.com/.

Abstract

Large language models are becoming part of everyday software engineering, but their security value and limitations need to be assessed in the context of real industrial workflows. This talk presents three perspectives from Siemens research and practice. First, we examine how LLMs can support the creation of more secure Dockerfiles in our ASE industry showcase paper. We then discuss the security challenges of deploying LLM-assisted software engineering in an enterprise: dependency confusion, prompt injection, and the potential leakage of sensitive source code, configuration, or business data. Finally, we use automatic configuration hardening of new Linux-based systems against Center for Internet Security (CIS) recommendations as a concrete example of a task that current LLM-based assistance does not yet perform reliably. The example is intended to spark discussion about why plausible advice is not the same as complete, verifiable, and safe automation, and where the limitations of today's LLMs become decisive.

Bio

Dr. Patrick Stöckle is a Research Scientist at Siemens Research. His research and industrial focus encompass mainly container security and configuration management which resulted in several ASE publications. He received his PhD in Computer Science from the Technical University of Munich (TUM). Before that he completed his undergraduate and master's studies in Computer Science at the Karlsruhe Institute of Technology (KIT).

Abstract

Large language models are reshaping how software is designed, built, tested, and secured. In this 90-minute session, Oliver Moeller (AWS Solutions Architect, Bundesliga) and Elmar Husmann (Principal Tech BD, AWS Global Education) show how agentic, spec-driven development with AWS Kiro turns natural-language intent into working software, and how it's already delivering value in professional sports. We open with real-world practice: how the Bundesliga uses Kiro for developing new digital fan experiences and sports-data analysis, and how over 320 developer teams from 41 countries used Kiro at the TUM Football Analytics Hackathon 2026 and the World Sports Innovation Cup with Adidas and Bundesliga to create innovation in sports. We then introduce the AI Software Development Lifecycle (AIDLC) method used by AWS, before moving into a hands-on workshop where we build an app on a real sports dataset, live. All participants receive free Kiro access for the duration of the LLM winter school; TU Munich students, and students from other eligible institutions, can keep building via the Kiro for Student program for 12 months. Bring a laptop.

Bio

Bio

Chunyang Chen is Professor of Software Engineering & AI at the Technical University of Munich. His research applies AI and related methods to mobile app development, software repositories, and software security. TUM profile