Abstract. Large language models have moved far beyond code completion. In this talk, I reflect on fifteen years of building search and AI systems at Microsoft — from Bing and Windows Search to Microsoft Copilot — and how the introduction of LLMs into the development cycle has fundamentally changed the way engineers and applied scientists work. The shift is not merely one of speed. It is a shift in what the scarce resource actually is: no longer the writing of code, but the understanding of complex systems, the quality of specifications, and the design of validation loops. I will explore how practices such as spec-driven development, agentic inner-loop testing, and LLM-assisted evaluation are changing the daily rhythm of engineering teams, and what this means for the skills, workflows, and culture that high-performing software organisations need to cultivate. Drawing on concrete examples from building and shipping AI at scale, I will argue that the most important question is not how to make engineers faster, but how to redesign the development process itself for a world in which code is increasingly cheap and human attention is the binding constraint.

Bio. Paul Baecke is a Partner Group Engineering Manager at Microsoft AI, based in London. Over 15 years at Microsoft, he has worked at the intersection of search, AI, and productivity — beginning with Bing and Windows Search, where he helped build relevance and ranking systems at global scale, and progressing to his current role leading a team of over 100 engineers and scientists working on Microsoft 365 Copilot, Microsoft's flagship AI assistant and the centrepiece of its vision for the future of work. Paul's team is responsible for the product quality, evaluation, and post-training of models. He holds multiple granted US patents in search, machine learning, and intelligent interfaces. Paul holds degrees in Mathematics, Applied Mathematics and Computer Science from Rhodes University and Philosophy from Tilburg University.

Abstract. This talk covers recent work on automated software testing at Meta, focussing on Just-in‑Time Tests (JiTTests).

JiTTests are a fundamentally novel approach to testing where tests are automatically generated by large language models (LLMs) on the fly to catch bugs – even ones that traditional testing might not catch – just-in-time before the code lands into production. A Catching JiTTest focuses specifically on finding regressions introduced by a code change. This type of testing reimagines decades of software testing theory and practice.

While traditional testing relies on static test suites, manual authoring, and ongoing maintenance, Catching JiTTests require no test maintenance and no test code review, meaning engineers can focus their expertise on real bugs, not false positives. Catching JiTTests use sophisticated techniques to maximize test signal value and minimize false positive drag, targeting test signals where they matter most: on serious failures.

Blog post: https://engineering.fb.com/2026/02/11/developer-tools/the-death-of-traditional-testing-agentic-development-jit-testing-revival/

arXiv version of forthcoming FSE 2026 paper: https://arxiv.org/abs/2601.22832

Bio. Mark Harman is a Research Scientist at Meta London and a professor at University College London. He joined Meta following acquisition of his startup Majicke. He has published over 300 papers, with over 48,000 citations, and an H index of 111, making him the most highly cited scientist in the field of both Software Testing and of Program Analysis. His work has been deployed throughout Meta’s platforms for the past nine years, directly impacting over 3 billion people who rely on its product’s for social networking, community building and communication. His work has also directly impacted more than 200 million small companies that use Instagram, Facebook and WhatsApp to reach their customers and indirectly impacted many others that have deployed technology based on it, such as Microsoft, Google and Amazon. For his scientific work, Harman received the IEEE Harlan Mills Award and the ACM Outstanding Research Award in 2019. In 2020, he was elected a fellow of the Royal Academy of Engineering.

Abstract. Test cases are a central artifact in software testing, used to validate system behavior, support regression testing, and document expected functionality. However, automatically generated test cases face two well-known challenges: they often suffer from poor understandability, and the automatic generation of test oracles remains difficult. This talk examines how Large Language Models (LLMs) can be leveraged to address these challenges. First, I will talk about the use of LLMs to improve the understandability of test code, focusing on transforming existing tests into clearer and more structured forms that are easier to interpret and maintain. Second, I will explore test oracle generation, including both expected-output oracles and more complex invariant-based oracles, highlighting how LLMs can assist in deriving expected behaviors in the absence of explicit specifications. I will also discuss the environmental cost of generating test artifacts with LLMs, comparing it to more traditional approaches to test generation, and consider the trade-offs involved when adopting such techniques.

Bio. Gunel Jahangirova is a Lecturer in Computer Science at King's College London. Before joining King’s, she worked as a postdoctoral researcher at Università della Svizzera italiana in Lugano, Switzerland. She obtained her PhD through a joint program between Fondazione Bruno Kessler in Trento, Italy, and University College London in the UK. Her research focuses on the automatic generation and evaluation of test oracles, error propagation in software systems, testing of deep learning systems, oracle design and quality metrics for autonomous vehicles, and the applications of AI to software engineering tasks. She has served on the program committees of major software engineering (ICSE, FSE, ASE) and software testing conferences (ISSTA, ICST), as well as for leading journals (TSE, TOSEM, EMSE). Her work has received IEEE Distinguished Paper Award in 2020. She is a member of the editorial board of the Empirical Software Engineering journal.

Abstract. While Large Language Models and agentic AI systems have demonstrated remarkable success on open-source program repair benchmarks, deploying these systems in a complex, enterprise-scale environment introduces a new set of distinct challenges. This talk explores recent research from Google’s DevAI team on bridging the gap between academic automated program repair (APR) and practical industrial deployment.

We will explore the ecosystem of agentic bug-fixing, centered around Passerine, an agentic repair system designed to operate within a massive, multilingual monorepo.

The talk will cover several critical areas of our research, including: (i) examining how real-world enterprise bugs differ from standard open-source benchmarks (like SWE-Bench) and why they require tailored agentic approaches; (ii) enabling agents to autonomously generate Bug Reproduction Tests (BRTs) and the benefits of dynamically cogenerating the fix and the test in a single trajectory; (iii) combating developer review fatigue by applying bug abstention (filtering unfixable bugs prior to execution) and patch validation (rejecting unlikely fixes); and (iv) utilizing LLMs-as-a-Judge alongside human-refined rubrics to reliably and scalably evaluate patch validity.

This talk is designed for software engineering researchers and practitioners interested in applied AI, large language models, automated testing, and program repair. Attendees will leave with a holistic understanding of how agentic software engineering is evolving to handle real-world complexity, establish ground-truth evaluations, and respect the cognitive load of human reviewers.

Bio. Michele Tufano is a Sr. Software Engineer on the DevAI Research team at Google, where he focuses on the intersection of Artificial Intelligence and Software Engineering. His current work centers on designing agentic systems for automated program repair, bug reproduction testing, and LLM-driven development at an industrial scale. Prior to joining Google, Michele was a Research Scientist at Microsoft, continuing his research on AI for Code. He holds a Ph.D. from William & Mary, with a thesis focusing on Neural Machine Translation for automated bug-fixing.

Abstract.

Issue reports, i.e., textual descriptions of bugs, feature requests, and other software-related issues, are a central artifact in modern software maintenance and are used to track, prioritize, and manage development tasks, as well as to support debugging, feature evolution, and team coordination. Classifying them according to their nature is both critical and time-consuming, and provides an ideal testbed for studying how LLMs perform in specialized text classification tasks, particularly given the challenges posed by long and often unstructured reports containing mixed information, the variability of model predictions across different runs and settings, and the need for outputs that are both reliable for practical use and interpretable by developers.

This talk will present novel methods to address these challenges grounded in two complementary perspectives: understanding user intent and example-guided decision-making. It will also introduce several methodological tools, including (i) a MapReduce-like approach to handle long reports and preserve relevant information, (ii) techniques for measuring LLM consistency across multiple runs and temperature settings with direct implications for reproducibility in empirical studies, and (iii) using perplexity as a confidence signal to trigger human review and develop more intelligent ensemble strategies. Along with exploring unexpected LLM behaviors and the unique misclassification causes in issue reports, the talk will offer specific recommendations for implementing these techniques in real- world scenarios where reliability, cost, and privacy are all important.

Bio. Gabriele De Vito is a postdoctoral researcher at the University of Salerno, where he received his PhD in Computer Science. With over twenty years of experience in software engineering, he has held roles including R&D Manager, CIO, and CTO, and has scientifically coordinated national and European-funded research projects with a total value exceeding €70 million. He has served on the program committees of international conferences (e.g., EASE, SEAA) and as a reviewer for software engineering journals (e.g., IEEE TSE). He co-chaired IWSM- MENSURA in 2023 and 2026. His work has been recognized with the COSMIC Research Award 2019. His research interests primarily lie in software engineering, machine learning, and large language models.

Abstract.

The integration of Large Language Models (LLMs) and autonomous agents into the software development lifecycle represents a fundamental shift in the socio technical landscape of software production. In this era, the traditional paradigms of quality and reliability are challenged by the broader concept of trustworthiness. This talk explores the multi-faceted challenges of AI-driven software engineering, examining the critical intersection of trustworthiness, legal liability, and ethics within AI-driven environments, moving beyond speculative benefits to analyze the empirical evidence of AI's impact on code quality and technical debt.

Bio. Fiorella Zampetti is an Assistant Professor at University of Sannio, Italy. She received her Ph.D. degree in Information Technologies for Engineering from the University of Sannio (Italy) in 2019. Her current research is focused on empirical software engineering, artificial intelligence for software engineering, DevOps for Cyber-Physical Systems (CPSs), software maintenance and evolution, and mining software repositories. She has served as a program committee member of international conferences in the software engineering field and she also serves as a reviewer for Software Engineering journals including TOSEM, TSE, EMSE, JSEP, and JSS. She is Program co-chair for the ICSME Visions and Emerging Results Track 2026 and also Program co-chair for TechDebt 2027. She is a member of the editorial boards of the Journal of Systems & Software and the Journal of Software: Evolution and Process, the Journal of Systems & Software, and Science of Computer Programming.