Abstract. With the proliferation of digital services and the growing number of cyber-attacks, there is increasing emphasis on adopting security and privacy by design paradigms for engineering software underpinning digital systems. Hence, it is important to empirically gain and share in-depth understanding of the technological, organisational, processes and practices that purport to support security and privacy by design paradigms. One of our main research goals is to empirically understand and support security and privacy by design paradigm for engineering of trustworthy digital services. To this end, we have conducted several studies that have enabled us to assess the current research in software security and privacy, identify important research problems and devise/evaluate appropriate solutions for helping practitioners in incorporating security intelligence and privacy protection mechanisms in their software development practices. Through this process, we encountered and/or observed several methodological challenges that needed appropriate strategies for producing robust and impactful research. In this tutorial, I’ll share some of the methodological challenges and strategies that can be expected in conducting evidence-based research for supporting software security based on our R&D efforts while discussing different aspects of our studies. I’ll also briefly touch upon the key lessons we have learned from applying empirical approaches in secure software development R&D.

Bio. M. Ali Babar is a Professor in the School of Computer Science, University of Adelaide, Australia. He leads a theme on architecture and platform for security as service in Cyber Security Cooperative Research Centre (CSCRC), a large initiative funded by the Australian government, industry, and research institutes. Professor Babar leads one of the largest projects on “Software Security” in the ANZEC region funded by the CSCRC. Software Security with Focus on Critical Infrastructure, SOCRATES, brings more than 75 researchers and practitioners from 10 organization for developing and evaluating novel knowledge and AI-based platforms, methods, and tools for software security. Prof Babar established an interdisciplinary research centre called CREST, Centre for Research on Engineering Software Technologies, where he leads the research, development and education activities of more than 30 researchers and engineers in the areas of Software Systems Engineering, Security and Privacy, and Social Computing. Professor Babar has authored/co-authored more than 300 peer-reviewed research papers at premier Software journals and conferences. Professor Babar obtained a Ph.D. in Computer Science and Engineering from the school of computer science and engineering of University of New South Wales, Australia. He also holds a M.Sc. degree in Computing Sciences from University of Technology, Sydney, Australia. More information on Professor Babar can be found in his website or Google Scholar.

Abstract. Large Language Models (LLMs) have gained significant attention in the software engineering community. Nowadays developers have the possibility to exploit these models through industrial-grade tools providing a handy interface toward LLMs, such as GitHub Copilot. In this talk, I will discuss recent successful applications of LLMs for code generation, showing how they are changing the way in which developers approach coding. Then, I will present open problems in the area of AI-based code generators; this part will cover both issues arising from their usage as well as challenges that the research community working in this area are facing (e.g., related to the empirical evaluation of these tools).

Bio. Gabriele Bavota is an associate professor at the Faculty of Informatics of the Università della Svizzera italiana (USI), Switzerland, where he is part of the Software Institute and he leads the SEART research group. He received the PhD in Computer Science from the University of Salerno, Italy, in 2013. His research interests include software maintenance and evolution, recommender systems, mining software repositories, and empirical software engineering. On these topics, he authored over 170 papers appeared in international journals and conferences and has received four ACM Sigsoft Distinguished Paper awards at the three top software engineering conferences: ASE 2013 and 2017, ESEC-FSE 2015, and ICSE 2015. He also received the best/distinguished paper award at SCAM 2012, ICSME 2018, MSR 2019, and ICPC 2020, and the MIP Award at SCAM 2022. He is the recipient of the 2018 ACM Sigsoft Early Career Researcher Award for outstanding contributions in the area of software engineering as an early career investigator and the principal investigator of the DEVINTA ERC Starting grant. More information is available here.

Abstract. Assured Large Language Model Software Engineering (Assured LLMSE) addresses the twin challenges:

1. Ensuring LLM-generated code does not regress the properties of the original code
2. Quantifying the improvement over the original archived by the improved code in a verifiable and measurable way.

In so doing, the Assured LLMSE approach tackles the problem of LLMs' tendency to hallucinate, as well as providing confidence that generated code improves an existing code base. Software testing and measurement play critical roles in this improvement process: testing is the guard against regression, while measurement provides the quantifiable assurance of improvement. Assured LLMSE takes its inspiration from previous work on genetic improvement, for which software measurement also plays a central role. In this keynote we outline the Assured LLMSE approach, highlighting the role of software measurement in the provision of quantifiable, verifiable assurances for code that originates from LLM–based inference.

This is joint work with Nadia Alshahwan, Andrea Aquino, Jubin Chheda, Anastasia Finegenova, Inna Harper, Mitya Lyubarskiy, Neil Maiden, Alexander Mols, Shubho Sengupta, Rotem Tal, Alexandru Marginean, and Eddy Wang.

Bio. Mark Harman is a full-time Research Scientist at Meta Platforms in the Instagram Product Performance team, working on software engineering automation. He was previously in the Simulation-Based Testing (SBT) team at Meta, which he co-founded. The SBT team developed and deployed both the Sapienz and WW platforms for client- and server- side testing. Sapienz grew out of Majicke (a start up Mark co-founded) that was acquired by Facebook (now Meta Platforms) in 2017. Prior to working at Meta Platforms, Mark was head of Software Engineering at UCL and director of its CREST centre, where he remains a part time professor. In his more purely scientific work, he co-founded the field Search Based Software Engineering (SBSE) in 2001. He received the IEEE Harlan Mills Award and the ACM Outstanding Research Award in 2019 for his work, and was awarded a fellowship of the Royal Academy of Engineering in 2020.

Abstract. Test cases represent the first line of defence against the introduction of software faults, especially when testing for regressions. They must be constantly maintained and updated as part of software components to keep them useful. With the help of testing frameworks, developers create test methods and run them periodically on their code. The entire team relies on the results from these tests to decide whether to merge a pull request or deploy the system. Unfortunately, tests are not immune to bugs or technical debts: indeed, they often suffer from issues that can preclude their effectiveness. Typical problems in test cases are called flaky tests and test smells.

Over the last decades, the software engineering research community has been proposing a number of static and dynamic approaches to assist developers with the (semi-)automatic detection and removal of these problems. Despite this, most of these approaches rely on expensive dynamic steps and depend on tunable thresholds. These limitations have been partially targeted through machine learning solutions that could predict test quality issues using various features, like source code vocabulary or a mixture of static and dynamic metrics.

In this tutorial, I will discuss our experience building prediction models to detect quality issues in test code. The tutorial will discuss the design choices to make in the context of test code quality prediction and the implications these choices have for the reliability of the resulting models.

Bio. Valeria Pontillo is a post-doctoral researcher at the Software Languages Lab of the Vrije Universiteit Brussel with Prof. Coen De Roover. She received the PhD in Computer Science from the University of Salerno, Italy, in 2024. Her research activities are in Software Engineering, particularly Software Testing, Software Quality, and Software Maintenance and Evolution. Her research interests also include the development of novel software engineering for artificial intelligence tools and techniques. She served as a reviewer for international conferences (e.g., SANER 2024, SCAM NIER, ICSME NIER, ICSE 2024 Artifact Evaluation) and journals in the software engineering field (e.g., EMSE, TOSEM, TSEm JSS, IST). More information is available here.