BeMuSSE National Project Awarded to Benchmark LLM-Based Multi-Agent Systems in Software Engineering

Wed, 05 Nov 2025 00:00:00 +0000

We are thrilled to announce the funding of our new national research project, BeMuSSE (Systematic Benchmarking of LLM-based Multi-Agent Systems for Software Engineering)! Funded through the 2024 Knowledge Generation Call by the Spanish Ministry of Science, Innovation and Universities and the State Research Agency, with co-funding from the European Union, the project will be led by our principal investigators Roberto Rodríguez Echeverría and José María Conejero Manzano. Through BeMuSSE, we aim to establish the definitive evaluation framework for the next era of AI-driven software engineering.

Towards Rigorous and Transparent Multi-Agent Evaluation

In recent years, software engineering has entered a period of rapid evolution. The field is moving past single-task AI assistants toward deploying complex LLM-based multi-agent systems (LMAs) capable of orchestrating full software development lifecycle (SDLC) workflows—from initial implementation to testing and refactoring. However, evaluating these multi-layered systems presents significant hurdles due to the interplay between LLM decision engines, orchestration layers, and external tools. Current evaluation practices often depend on oversimplified pass rates alongside opaque or unstandardized benchmarks, neglecting core engineering priorities such as resilience, security, operational costs, and ethics.

To address this gap, BeMuSSE operates on a core premise: systematically applying established software engineering bodies of knowledge to AI-automated workflows, building upon proven standards such as ISO/IEC 25010, ISO/IEC 12207, and OWASP recommendations. Accordingly, BeMuSSE proposes a multidimensional conceptual framework structured around four fundamental pillars:

Functional Accuracy: Guaranteeing that generated code satisfies technical requirements, handles edge cases, and integrates seamlessly without errors.
Orchestration & Process: Evaluating decision traces, transparency, and collaborative efficiency across autonomous agents as well as hybrid human-agent teams.
Non-Functional Quality: Analyzing security vulnerabilities, environmental sustainability, execution costs, and system robustness.
Ethics & Compliance: Verifying adherence to regulatory standards, organizational policies, and the active mitigation of AI-induced bias.

The centerpiece of this strategy will be conceptual modeling for multi-agent architectures. By extracting entities and relationships from domain knowledge, the project team will develop a dedicated domain-specific language (DSL) explicitly tailored for LMA benchmarking.

Over its three-year timeline, BeMuSSE will progress from foundational applied research to a validated prototype tool deployed in realistic test environments. The project methodology combines the European Commission’s PM² management framework with agile engineering practices distributed across six distinct work packages.

Key deliverables will include open-source evaluation tooling, a public leaderboard tracking LMA performance, and actionable ethical guidelines to support responsible industry adoption.

BeMuSSE goes well beyond a standard academic enterprise. In today’s fast-moving software landscape, establishing clear ways to evaluate, verify, and trace increasingly complex AI agents is essential. Ensuring that AI-generated software remains dependable, transparent, and ethical is precisely the challenge BeMuSSE is set to tackle, and we look forward to leading this initiative.

Curious about our research areas, interested in learning more about our ongoing initiatives, or looking to collaborate? Reach out to our team!

National Project | i3 lab

BeMuSSE National Project Awarded to Benchmark LLM-Based Multi-Agent Systems in Software Engineering

Towards Rigorous and Transparent Multi-Agent Evaluation