July 2026 – SE metrics (Software Engineering)

Summer 2026 – where are we going now?

It’s hard to believe that 2026 is already halfway and the summer is upon us. When I look back at what happens now, I still think that the best time to be a software engineer is now. We get so many cool tools to work with that we do not want to stop working. The technology is becoming more powerful and our expertise is needed increasingly often.

First, we need to design software. Maybe I’m biased because of the topic of my PhD, but this is what I really like doing: getting dirty with the code and the design. Now, I can do more of that, reducing my frustrations with defects to zero. Why should I care? Claude and Codex can fix that for me! Ok, I do exaggerate, they are not perfect and sometimes I just need to rewrite the code myself, but hey, I can still do it.

Second, we need to define intents and specifications. That part is not my favorite because it means more requirements. Requirements are not something that I particularly like. There is nothing wrong with them; it’s just not my cup of tea (or coffee, choose the beverage of your liking). Now, I can work with requirements a bit more because I have the AI to help me with them. I do not need to work so much with the form; I can focus on what I want. So, I’ve learned meta-prompting and meta-specifications, which are really cool.

Third, we need to create guardrails – functional, development, non-functional. I’ve worked on a cool project together with my students and one of the companies, which opened up my eyes to just how important guardrails are. I’ve also realized how cool it is to work with them – it’s like setting prerequisites for learning for someone else, but that someone else is software. Take a look at our example and, if you like it contribute: https://github.com/miroslawstaron/elevator_runtime_guardrail_simple

Finally, we can learn faster because the new tools provide us with the right support for that. No more StackOverflow or community questions, we have Gemini, Claude, ChatGPT, Antigravity, you name it, that help us learn new things. They can even write things for us in the right format – you like a book chapter, here it comes; you like a paper, no problem at all; or maybe just a PowerPoint with some examples, we can do that too.

I’m looking forward to what the next year brings and how we will tackle new challenges. OpenAI usually releases things in the summer, so I hope to see something really new and cool now.

Have a great summer!

Levels of automated code development…

Image generated by Gemini based on this blow post

https://www.mdpi.com/2076-3417/16/10/4788

The practical meaning of automated code generation is shifting rapidly. What was recently categorized as simple “autocomplete” has expanded into complex workflows involving multi-file modifications, test execution, and repository navigation. However, as Zhenhan Chen et al. argue in a recently published article in Applied Sciences, the software engineering community still lacks a shared, operational language to describe exactly how much work is being delegated to these AI systems.

To fill this critical gap, the researchers proposed “Levels of Automated Code Generation” (LACG), a six-level taxonomy (L0 to L5) designed to classify the degree of automation in AI-augmented software construction.

The proposed levels include:

L1 (Assisted Generation): Localized, token-level assistance (e.g., inline completion) with full human fallback.
L2 (Partial Generation): Generation of complete code units (e.g., functions, classes) from prompts, still requiring human integration and verification.
L3 (Conditional Automation): The system executes multi-step tasks (e.g., bug fixes, feature implementation) within a constrained OCD. The human is the final fallback.
L4 (High Automation): The system autonomously manages end-to-end development of subsystems within a broader OCD and owns the operational fallback.
L5 (Full Automation): Unrestricted software engineering across any domain, owning all fallback and recovery duties.

Conclusion

The LACG taxonomy provides a disciplined, operational vocabulary necessary for future empirical work, benchmark design, and reasoning about responsibility allocation in AI-augmented coding. While the study demonstrates the framework’s applicability, the authors clarify that it does not serve as a prediction of tool performance, security, or productivity outcomes

What are you talking about – one agent asked another…

Image taken directly from the paper

https://arxiv.org/pdf/2605.24138

The Software Engineering (SE) landscape is shifting from LLM-assisted workflows, like copilots, toward Autonomous SE, where multiple specialized AI agents cooperate without a human in the loop. The premise is exciting: a ‘Designer’ agent creates the plan, and a ‘Programmer’ agent implements it. Yet, simply letting agents talk to each other does not reliably lead to correct or stable solutions. In our new paper, my colleagues and I undertake a systematic analysis to understand why.

We explored conversations between a Designer and a Programmer across 12 combinations from 7 leading open-source models—including Gemma 2/3, LLaMA 3.2/3.3, Qwen3, and the reasoning-focused DeepSeek-R1—as they tried to build a mathematical game in C (Fibonacci). We found that the interactions are complex, non-linear, and prone to surprising failures.

Echo Chambers Instead of Collaboration

One of our most critical, blog-worthy findings is that common metrics used to measure conversational “success,” like high BLEU and ROUGE scores (which track semantic alignment), can be misleading. In mismatched pairs, particularly involving non-reasoning models (like Gemma 3 or MiniCPM) paired with reasoning models (DeepSeek-R1), high scores were a red flag for “semantic echoing.” The Programmer agent simply mirrored the Designer’s output verbatim, which was a conversational failure, not a collaborative victory.

DeepSeek-R1: The Lone Convergent Pair

In terms of actual solution correctness, the results were stark. Only a single agent pair—DeepSeek-R1 paired with itself—was unique in immediately converging to the correct solution and sustaining it consistently to the final iteration. This indicates that while reasoning capabilities are crucial, stable collaboration currently depends more on consistent role conditioning. Our manual inspections showed that DeepSeek-R1:DeepSeek-R1 prioritized design discussion over echoing, which contributed to its success despite having some “No Code Found” instances, often related to compilation instructions.

Talking Themselves Out of Success: The Threat of Drift

We also identified a critical trend we call “behavioral stagnation” and “drift.” Multiple promising pairs—including Qwen3:DeepSeek-R1, DeepSeek-R1:LLaMA 3.3, and even a same-model pair, LLaMA 3.3:LLaMA 3.3—actually started with the correct solution. However, they subsequently talked themselves out of it, diverging to other topics (like related number theories or other code snippets) and never converging again.

As we noted in our analysis, late recovery from this kind of drift is unlikely. This provides an essential behavioral signal for SE tools developers: you must monitor the health of the interaction trace (for repetition, topic drift, or role instability) in real-time, rather than relying solely on whether code is eventually produced. Monitoring these conversational patterns can inform early stopping conditions or trigger prompt revisions before computational time is wasted on non-productive exchanges.

The Takeaway

As Software Engineering transitions to autonomous agent teams, understanding and calibrating these multi-agent interaction dynamics is critical. Strong semantic alignment does not ensure correctness, and reasoning capability alone does not guarantee stable collaboration. You need clear role separation, pair compatibility, and robust monitors that can detect conversational drift. Success isn’t a final code snippet; it’s a healthy conversation.