LLM | i3 lab

SISTEDES 2026 Alicante Conference

Fri, 19 Jun 2026 00:00:00 +0000

From June 16 to 18, Roberto, Juan Diego (Andy), Emilio, and Javier attended the SISTEDES 2026 Conference in Alicante, held between the Aula de Cultura of the Fundación Mediterráneo and the Sede Universitaria Ciudad de Alicante. The event brought together the software engineering community around several tracks, with a special interest for us in AI4SE.

Our participation began on Tuesday, the 16th, with the session dedicated to automatic improvement and repair. There, we presented the short paper Quality Assessment of Agentic AI Processes in Software Development Life Cycle, focused on the evaluation and comparison of traces generated by agentic processes in software development. By evaluating the agent’s reasoning traces, the correctness of the process can be formally verified, ensuring it meets the established quality requirements.

That same day, we attended other sessions that, as a whole, provided a good overview of the current state of research in automation, static analysis, and the use of LLMs for maintenance and validation tasks.

Tuesday’s keynote was delivered by Nuria Oliver, with an intervention focused on the role of AI in society, its opportunities, and its limits, through the lens of the ELLIS Alicante’s work. It was a clear and useful talk that raised important questions about AI biases and limitations from a more human perspective.

Wednesday, the 17th, concentrated several of the most relevant interventions. Jordi Cabot’s keynote on BESSER took place, a proposal aimed at accelerating software development through modeling and AI collaboration. It was a particularly interesting presentation due to how it combines model-driven engineering, automation, and generative tools. Following this, Roberto collaborated on the presentation of the Commission for Teaching Innovation in Software Development Education in the Era of Generative AI, which addressed the impact of generative tools on university teaching. The main proposal of this talk was to gather different teaching innovation experiences with generative tools into a common repository, so that it serves as a reference for other teachers and facilitates collaboration among them.

Later that same afternoon, Emilio presented Beyond Functional Code: Quality Specifications for LLM-based Agents, a proposal focused on separating functional specifications from quality ones so that the latter can be reused independently of the technological stack. The idea relies on a workflow that converts information obtained from static analysis tools into reusable quality specifications, aiming to reduce technical debt and advance toward a model applicable to different software systems.

Thursday, the 18th, closed the conference with a keynote by Verónica Dahl, a Prolog pioneer, who offered a historical and technical look at logic programming and some of its most interesting formal developments, as well as its use in teaching younger students.

The closing session on the final day was particularly relevant to us, as Juan Manuel Murillo, coordinator of our research group (Quercus-SEG), was unanimously appointed as the new president of the society. We are all looking forward to this new chapter for SISTEDES.

Social activities played a secondary role but allowed us to converse with other researchers and strengthen contacts. In any case, the main value of the trip lay in the quality of the presentations and the ideas that emerged to continue working.

Conclusions

SISTEDES 2026 was particularly useful for situating our work within the context of current research and for learning firsthand about advancements in the use of LLMs and other AI techniques in software engineering. Interacting with other researchers and being exposed to different approaches provided us with new perspectives and opportunities for collaboration.

If you’d like to learn more about our research lines in applying AI to software engineering, feel free to get in touch with us and we’ll share more details.

BeMuSSE National Project Awarded to Benchmark LLM-Based Multi-Agent Systems in Software Engineering

Wed, 05 Nov 2025 00:00:00 +0000

We are thrilled to announce the funding of our new national research project, BeMuSSE (Systematic Benchmarking of LLM-based Multi-Agent Systems for Software Engineering)! Funded through the 2024 Knowledge Generation Call by the Spanish Ministry of Science, Innovation and Universities and the State Research Agency, with co-funding from the European Union, the project will be led by our principal investigators Roberto Rodríguez Echeverría and José María Conejero Manzano. Through BeMuSSE, we aim to establish the definitive evaluation framework for the next era of AI-driven software engineering.

Towards Rigorous and Transparent Multi-Agent Evaluation

In recent years, software engineering has entered a period of rapid evolution. The field is moving past single-task AI assistants toward deploying complex LLM-based multi-agent systems (LMAs) capable of orchestrating full software development lifecycle (SDLC) workflows—from initial implementation to testing and refactoring. However, evaluating these multi-layered systems presents significant hurdles due to the interplay between LLM decision engines, orchestration layers, and external tools. Current evaluation practices often depend on oversimplified pass rates alongside opaque or unstandardized benchmarks, neglecting core engineering priorities such as resilience, security, operational costs, and ethics.

To address this gap, BeMuSSE operates on a core premise: systematically applying established software engineering bodies of knowledge to AI-automated workflows, building upon proven standards such as ISO/IEC 25010, ISO/IEC 12207, and OWASP recommendations. Accordingly, BeMuSSE proposes a multidimensional conceptual framework structured around four fundamental pillars:

Functional Accuracy: Guaranteeing that generated code satisfies technical requirements, handles edge cases, and integrates seamlessly without errors.
Orchestration & Process: Evaluating decision traces, transparency, and collaborative efficiency across autonomous agents as well as hybrid human-agent teams.
Non-Functional Quality: Analyzing security vulnerabilities, environmental sustainability, execution costs, and system robustness.
Ethics & Compliance: Verifying adherence to regulatory standards, organizational policies, and the active mitigation of AI-induced bias.

The centerpiece of this strategy will be conceptual modeling for multi-agent architectures. By extracting entities and relationships from domain knowledge, the project team will develop a dedicated domain-specific language (DSL) explicitly tailored for LMA benchmarking.

Over its three-year timeline, BeMuSSE will progress from foundational applied research to a validated prototype tool deployed in realistic test environments. The project methodology combines the European Commission’s PM² management framework with agile engineering practices distributed across six distinct work packages.

Key deliverables will include open-source evaluation tooling, a public leaderboard tracking LMA performance, and actionable ethical guidelines to support responsible industry adoption.

BeMuSSE goes well beyond a standard academic enterprise. In today’s fast-moving software landscape, establishing clear ways to evaluate, verify, and trace increasingly complex AI agents is essential. Ensuring that AI-generated software remains dependable, transparent, and ethical is precisely the challenge BeMuSSE is set to tackle, and we look forward to leading this initiative.

Curious about our research areas, interested in learning more about our ongoing initiatives, or looking to collaborate? Reach out to our team!

International Summer School on LLM-based Agents for Software Engineering

Tue, 16 Sep 2025 00:00:00 +0000

From September 1st to 3rd, 2025, i3Lab (Quercus Software Engineering Group and INTIA, Universidad de Extremadura) successfully hosted the 1st International Summer School on LLM-based Agents for Software Engineering (LLMA4SE). The event brought together researchers, practitioners, and students from academia and industry to explore the fast-evolving field of LLM-based multi-agent systems and their applications in software engineering.

The opening day included two interesting talks:

AgentWare in Software Engineering: A Taxonomy of LLM-Based Multi-Agent Systems by Davide Di Ruscio, which led to stimulating discussions.
Generative AI Models Running in Your Own Infrastructure by Jesús M. González Barahona, who offered a deep perspective on openness in AI models.

Participants also enjoyed a hands-on workshop on TDD with the Help of AI, led by Jesús M. González Barahona and Giovanni Rosa. The day concluded with a cultural highlight: a guided tour of the UNESCO World Heritage city of Cáceres.

The second day offered a strong mix of research and practical applications. The day began with three particularly engaging presentations:

Multi-agents for Code Generation by Jie Zhang, addressing current challenges and approaches.
AI and Software Development: An Industry Paradigm Shift by Mario Corchero Jiménez and David Naranjo, presenting the industry perspective.
From Workflow-based to Fully Agentic Applications: smolagents and LangGraph by Antonio García Domínguez, who introduced agent development frameworks.

In the afternoon, Antonio García Domínguez guided participants through a highly practical workshop on developing agentic applications with human-in-the-loop via LangGraph. The day also included two enriching social activities: a guided visit to the Helga de Alvear Museum of Contemporary Art and a social dinner at Gran Hotel Don Manuel.

The final day, we enjoyed some thought-provoking sessions:

Olivier Perard on AI in Enterprise Architecture.
Marc Alier on Safe AI in Education - The LAMB Project, focusing on history, ethics, and risks.
Merve Astekin on Software Maintenance with Agentic LLMs, showcasing sustainable AI projects.

The summer school concluded with an advanced workshop, Multi-Agent LLMs and Knowledge Graphs for Monitoring and Sustaining Software Systems, delivered by Adela N. Videsjorden and Simeon Andersen Tverdal.

With the success of this first edition still fresh in our minds, we are already hard at work planning the next one. We fully expect the second edition of our school to be just as attractive and insightful as this inaugural event. Our goal is to maintain the high quality of content and the variety of the discussions to create a great meeting point for the community.

Stay tuned for future announcements!