Large Language Models (LLMs) are revolutionary for programming productivity, producing functional code snippets in seconds. However, as software engineers, my co-authors and I know that “functional” is not the same as “well-designed.” LLMs are generally “bottom-up” thinkers; they excel at local syntax but struggle to adhere to higher-level architectural structures or design patterns, which are crucial for long-term software maintainability and scalability.
In our new paper, presented at PROMISE ’26, we set out to answer a critical question: How can we best guide LLMs to incorporate design patterns into their generated code without sacrificing functional correctness?
We decided to use the standard Singleton creational pattern as our case study due to its easily identifiable predicates. We designed a computational experiment evaluating 13 state-of-the-art LLMs (including GPT-4o Mini, Llama 3.3, and Qwen 3) across 164 Java coding challenges from HumanEval-X. We tested four distinct prompting strategies: simple natural language instructions, iterative binary automated feedback (“Is it Singleton? Yes/No”), extensive automated feedback identifying exactly which Singleton properties were missing, and extensive feedback combined with few-shot examples.
Our findings reveal that there is no one-size-fits-all prompting solution; the optimal strategy is highly model-dependent. However, a major takeaway is that even simple strategies work remarkably well. Overall, iterative binary feedback provided the best balance, maximizing alignment with the Singleton pattern while preserving or even improving the code’s functionality.
Surprisingly, enforcing design principles didn’t always hurt performance. For strong models like Llama 3.3, just instructing it to use Singleton resulted in 100% pattern adherence and actually increased functional test pass rates by 34 percentage points compared to the baseline.
Our study proves we can teach LLMs good design habits using automated feedback loops. You can read the full paper and access our experimental data here https://arxiv.org/pdf/2605.26898.
Getting Large Language Models (LLMs) to write functional code often feels like casting spells; a slight misphrasing in your prompt can result in a buggy output. This is even more important now that we have agents which work for days on our tasks.
The core issue is that while LLMs are powerful, their code generation performance is highly sensitive to prompt formulation. Traditional manual engineering is tedious, and existing automated techniques often treat prompt modifications—like lexical edits or semantic rewriting—in isolation. They also typically rely on binary (pass/fail) signals, ignoring valuable information about partial correctness.
When I was at VECS, I got to meet that Swedish Champion in prompting. He told me that the best technique is to use LLMs to create prompts. This paper embraces that idea and goes even further – creating a full reinforcement learning framework to make prompts.
In this paper, the agent is guided by shaped rewards derived from unit-test feedback. Instead of just rewarding full passes, the system provides denser learning signals by rewarding the proportion of test cases passed. This enables the agent to discover sequences of transformations that progressively improve the functional correctness of the generated code.
The framework was evaluated on a few widely known benchmarks (MBPP+, HumanEval+, APPS) using three code generators: CodeT5+, CodeLLaMA, and DeepSeek-Coder. On the MBPP+ test set (500 tasks), the PPO agent achieved strict Pass@1 scores of:
57.58% for CodeT5+
64.80% for CodeLLaMA
85.50% for DeepSeek-Coder
These results significantly outperformed direct generation and existing iterative strategies like EPIC and Reflexion. Furthermore, comparison against a “Random-Hybrid” baseline confirmed that the gains aren’t just from having the transformation tools, but from the agent learning how to intelligently schedule them based on feedback.
The key takeaway is clear: feedback-driven, multi-step RL optimization can move code generation beyond manual prompt engineering, providing an adaptive, automated path to functionally correct code.
Image source: Gemini, based on the summary of this blog post.
When I write this post, I’m sitting at a reporting workshop of Software Center, at Axis Communications in Lund. Jan has reminded us that we’ve been going on for 15 years. That’s most of my academic career and a lot of my life. Although it makes me feel old, let me reflect on what has happened and what will happen. After all, I need to live up to the nickname that my colleagues gave me – a dinosaur.
We actually started way earlier with a smaller initiative called Software Architecture Quality Center, which was only with Ericsson and the IT University of Gothenburg. In 2010, we realized that more companies need to join to make the collaboration more fruitful. It was nice, but let’s focus more on technology rather than people.
2010 was a period of rapid data growth, driven mostly by the introduction of the iPhone three years earlier. This means that we had to develop methods to rapidly develop software, so we had three themes: CI/CD – focused on processes and fast development, Architectures – focused on the structure of the software, and Metrics (where I was/still am) – focused on monitoring of quality, structure, and processes. Our focus resulted in several innovations, like Eiffel, using heatmaps, and efficient defect prediction at member companies.
Around 2015, we shifted our focus to data and began working on learning systems. Around 2020, we focused more on AI and machine learning, just as Jensen said, “AI is going to eat software.” Then, today, we see that we focus on advanced software – autoevolving systems, no architectures, multi-agentic AI systems, basically focusing on Software Engineering 4.0 or even beyond that.
I’ve analyzed the publications from Software Center, and here is what they look like.
Transition from agile development to continuous deployment
Software architecture and architecture evolution
Embedded systems software architecture
Architecture decisions
Architecture evolution and long-term maintainability
Software product lines and variability
Product-line engineering
Variability management
Legacy software product lines
Embedded and automotive software
Automotive software complexity and coupling
ISO 26262-related verification and validation
Automotive/telecom defect prediction
Requirements engineering
Experience-based requirements tools
Requirements clarification
Natural-language requirements categorization
Software ecosystems
Automotive ecosystems
Cross-organizational modeling
Software ecosystem workshops and coordination
Character of the period: This period is dominated by now classical software engineering themes: agile transformation, architecture, product-line engineering, embedded systems, and requirements. Continuous deployment appears, but mostly as an emerging transition target.
Agile research collaboration and organizational challenges
Automotive and embedded software
Automotive embedded requirements
Virtual verification ecosystems
Model use in automotive engineering
Controlled experimentation and A/B testing
Online controlled experimentation
Continuous experimentation
Experimentation at scale
Measurement and quality management
Measurement programs
Metrics for software design and architecture
Quality management under fast release cycles
Character of the period: The focus shifts from adopting agile to industrializing speed: CI/CD, DevOps, continuous experimentation, quality measurement, and technical debt become central. Automotive remains a strong application domain.
2020–2025: AI/ML systems, MLOps, federated learning, data pipelines
Main topics:
Machine learning and AI-enabled systems
Machine-learning systems engineering
AI for software analytics
ML-based test selection
ML pipelines and continuous delivery for ML systems
MLOps
MLOps frameworks
Maturity models
Trade-offs in MLOps adoption
Moving from ad hoc ML operations to systematic improvement
Federated learning
Federated learning architectures
Real-time end-to-end federated learning
Automotive federated learning case studies
Data pipelines and data-driven development
Data pipeline management
Data science driven processes
Continuous delivery for data/ML systems
Automotive software and software-intensive embedded systems
Automotive software architectures
Automotive A/B testing
Software-intensive embedded systems
Continuous deployment in embedded contexts
Testing and quality assurance
Exploratory testing
Test selection
A/B testing with limited samples
Testing in CI/CD pipelines
Requirements engineering for large-scale and automotive systems
Requirements engineering challenges
Balancing alignment and diversity of practices
Large-scale agile requirements practices
Technical debt and developer experience
Technical debt management
Developer morale
Incentives for technical debt reduction
Character of the period: This is the clear transition into AI/ML-oriented software engineering. The publication set moves from DevOps/continuous delivery for traditional software toward MLOps, federated learning, ML pipelines, AI-enabled systems, and data-driven organizations.
2025 onwards: Generative AI, LLMs, AI-assisted SE, automotive perception, ethics
Main topics:
Generative AI and AI for software engineering
Generative AI in automated software engineering
Hybrid classical-AI systems for testing and bug fixing
AI-enhanced experimentation
Large language models
Design pattern recognition using LLMs
LLM-generated graph/Cypher queries
Programming-language models
MLOps and continuous learning
MLOps adoption frameworks
Replay-based continuous learning
ML pipeline evolution
Automotive perception and vulnerabilities
Automotive software vulnerabilities
ML-based automotive perception systems
Data leakage detection for automotive perception
Ethics and requirements engineering
Ethics-driven requirements engineering
Autonomous vehicle guidelines
Cognitive biases in requirements engineering
Experimentation platforms and ecosystems
Extensible experimentation platforms
A/B test analysis at scale
Experimentation challenges in large product/service organizations
Cloud and IoT data architectures
AWS cloud data storage architectures
IoT data storage architecture comparisons
I think that we live in the most interesting times, especially as software engineers. We can focus on really cool things like innovation, ideation, and understanding domains, rather than learning exactly how pointer operations in C work. Well, I exaggerate a bit, as we still need to know what points do and how they work – and yes, if you use Rust, you still need to understand how the operating system works with the memory.
The future
In my view, the future will bring more software, better software, and more automation. Software engineers will focus on building platforms and APIs, creating guardrails, and deploying the software. We may need to get out of our comfort zone to actually talk to people, talk to our customers, and maybe even suppliers. We will constantly learn new things; AI will help us with that, and we will get better at creating more value from software than we do today.
It’s not just a dream, but a reality. OpenAI, Anthropic, and Google were started by just a few individuals. Now, we can even grow companies with the help of AI. Software Center has a mission to accelerate the adoption of new technologies, so let’s focus on the coolest of them all – Generative AI Multi-Agent Systems.
We have all seen Large Language Models (LLMs) write impressive snippets of code or debug a tricky function. AI coding editors like GitHub Copilot are increasingly adopted, with studies suggesting that up to 88% of developers report increased productivity.
But accelerations in development come with trade-offs. Existing studies have shown that LLMs often misuse APIs, introduce security vulnerabilities, and hallucinate. So I got to wonder: Can an LLM actually understand the soul of a complex software project? Can it generate a fully automated, high-quality commit (patch) that satisfies requirements and can be directly incorporated into a major production codebase?
This paper puts this question to the test, because it uses actual commits from substantial, real-world open-source systems. The authors developed an automated framework to assess how suitable LLMs are at fixing bugs and adding new features to sizable code bases. They applied this framework to 212 actual commits across eight popular open-source projects—including FFmpeg and wolfSSL—and three LLMs: GPT-4o, Ministral3-14B, and Qwen3-Coder-30B.
The framework tested the generated patches on three levels:
Verification: Does the generated code compile?
Validation (Static Analysis): Does it pass Clang’s static analysis checkers (e.g., memory safety checks)?
Validation (Dynamic Testing): Does it pass the project’s existing test suite?
The success rate varied wildly—from 0% on certain projects up to 60% on others. But overall, the verdict was clear: LLMs are not at a point where they can be effective contributors to production code. They still hallucinate, and they still have large limitations – at least the ones tested, we’ll see what the newest ones could do.
The Takeaway for Architects and Developers
The bottom line is clear: Do not trust LLMs de novo with critical production system code. They are effective for small functions, feature improvements, routine algorithms, and tasks similar to those seen in their training data.
However, the risk of “silent failures,” new security vulnerabilities, and logic regression means that rigorous human validation remains the most important step when integrating AI-generated contributions. Still, even in 2026!
