{"id":1020,"date":"2026-05-18T14:04:48","date_gmt":"2026-05-18T13:04:48","guid":{"rendered":"https:\/\/metrics.blogg.gu.se\/?p=1020"},"modified":"2026-05-13T14:09:29","modified_gmt":"2026-05-13T13:09:29","slug":"the-synthetic-engineer-measuring-the-real-impact-of-ai-on-software-delivery","status":"publish","type":"post","link":"https:\/\/metrics.blogg.gu.se\/?p=1020","title":{"rendered":"The Synthetic Engineer: Measuring the Real Impact of AI on Software Delivery"},"content":{"rendered":"\n
\n
\"\"<\/a><\/figure>\n<\/figure>\n\n\n\n

https:\/\/miroslawstaron.github.io\/hallucinations.html#\/5<\/a><\/p>\n\n\n\n

The shift from manual coding to AI-augmented orchestration is no longer a future – it is a reality. Software engineers adopt AI increasingly often and increasingly deep.<\/p>\n\n\n\n

However, as organizations pour investment into Generative AI tools, a critical question remains: How do we measure the true return on investment?<\/strong><\/p>\n\n\n\n

I asked Gemini to analyze the DORA report and look at the internet to find how people measure AI adoption. Its report, Evaluating the Synthetic Engineer<\/em>, suggests that we must move beyond vanity metrics like “lines of code generated.” When code generation is cheap, we need to think about the adoption and design.<\/p>\n\n\n\n

I’ve recently heard that one company paid an eqiovalent of three software engineers worth of tokens to Anthropic, for a seven-person team. This means that effectively, 30% of the entire team (3+7) was AI. This is really cool and it shows that this reality is here. How do we measure that these tokens were not just wasted, though?<\/p>\n\n\n\n

The Velocity-Quality Tension<\/h3>\n\n\n\n

The most immediate effect of AI is a spike in velocity. Teams often see a 15\u201325% reduction in Cycle Time<\/strong> and significantly accelerated onboarding\u2014reducing the “Time to 10th PR” from 91 days to just 33.<\/p>\n\n\n\n

However, this speed comes with a hidden cost: Comprehension Debt<\/strong>. The report highlights that AI-assisted code often results in higher defect density and a rework rate that can double the human baseline. To manage this, we must align AI metrics with the industry-standard DORA metrics<\/strong> to ensure that speed doesn’t break the system.<\/p>\n\n\n\n

Integrated Metrics Framework<\/h3>\n\n\n\n

To truly evaluate the AI organizations should track a mix of telemetry-based system data and survey-based human sentiment.<\/p>\n\n\n\n

Category<\/th>Metric<\/th>Measurement Source \/ Context<\/th><\/tr>
DORA (System)<\/strong><\/td>Deployment Frequency<\/td>CI\/CD Pipeline \/ Release logs<\/td><\/tr>
DORA (System)<\/strong><\/td>Lead Time for Changes<\/td>Version Control \/ Deployment logs<\/td><\/tr>
DORA (System)<\/strong><\/td>Change Failure Rate<\/td>Incident Management \/ CI\/CD logs<\/td><\/tr>
DORA (System)<\/strong><\/td>Recovery Time (MTTR)<\/td>Incident Management \/ Pager logs<\/td><\/tr>
AI Use<\/strong><\/td>Acceptance Rate<\/td>IDE Plugin Telemetry<\/td><\/tr>
AI Use<\/strong><\/td>AI Interaction Time<\/td>Tool Telemetry \/ Browser logs<\/td><\/tr>
AI Effect<\/strong><\/td>Rework Rate<\/td>Jira \/ Commit history<\/td><\/tr>
Human<\/strong><\/td>Trust & Reliance<\/td>Developer Surveys (Confidence in AI)<\/td><\/tr>
Human<\/strong><\/td>Job Satisfaction<\/td>Developer Surveys (Burnout vs. Flow)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Now, we can compare that to the DORA metrics that are used widely in industry today. There, we have two parts, the telemetry based ones: <\/p>\n\n\n\n

Metric<\/td>Definition<\/td>Measurement Source<\/td><\/tr>
Deployment Frequency<\/td>How often the team successfully releases to production.<\/td>CI\/CD Pipeline \/ Release logs<\/td><\/tr>
Lead Time for Changes<\/td>Time from code commit to code successfully running in production.<\/td>Version Control \/ Deployment logs<\/td><\/tr>
Change Failure Rate<\/td>% of deployments causing a failure in production (requiring a fix\/rollback).<\/td>Incident Management \/ CI\/CD logs<\/td><\/tr>
Failed Deployment Recovery Time<\/td>How long it takes to restore service after a failure in production.<\/td>Incident Management \/ Pager logs<\/td><\/tr>
Rework Rate<\/td>The percentage of work time spent on unplanned fixes or bugs.<\/td>Ticket tracking (Jira) \/ Commit history<\/td><\/tr>
Acceptance Rate<\/td>The ratio of AI-generated code suggestions that are actually kept in the file.<\/td>IDE Plugin Telemetry<\/td><\/tr>
Commit\/PR Volume<\/td>The raw count of code changes and pull requests submitted.<\/td>Version Control Systems (VCS)<\/td><\/tr>
AI Interaction Time<\/td>The actual duration of time spent interacting with an AI interface.<\/td>Tool Telemetry \/ Browser logs<\/td><\/tr>
Code Stability<\/td>The frequency of breaks or regressions in the automated test suite.<\/td>Testing Frameworks \/ Build logs<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

And then the ones that are measuring perceptions, based on surveys:<\/p>\n\n\n\n

Metric<\/td>Definition<\/td>Context for Use<\/td><\/tr>
Trust<\/td>The degree of confidence a developer has in the accuracy and safety of AI output.<\/td>To identify if developers are “blindly” following AI or if skepticism is hindering adoption.<\/td><\/tr>
Reflexive Use<\/td>How instinctively a developer turns to AI when a new problem arises.<\/td>To measure the behavioral shift in problem-solving habits.<\/td><\/tr>
Reliance<\/td>The self-assessed level of dependency on AI tools to complete daily work.<\/td>To monitor for potential skill atrophy or high-dependency risks.<\/td><\/tr>
Individual Effectiveness<\/td>Perceived productivity, impact on the organization, and ability to stay “in flow.”<\/td>To assess the “value-add” from the developer’s own perspective.<\/td><\/tr>
Job Satisfaction<\/td>The level of fulfillment and contentment a developer feels in their role.<\/td>To ensure that AI automation is improving work life rather than creating “toil.”<\/td><\/tr>
Burnout<\/td>Physical or mental exhaustion caused by work-related stress.<\/td>To monitor if the increased “instability” caused by AI is taxing the team.<\/td><\/tr>
Personal Ownership<\/td>The psychological feeling of “owning” the code and its quality.<\/td>To prevent the dilution of accountability when AI generates a high volume of code.<\/td><\/tr>
User-Centric Focus<\/td>The extent to which the team prioritizes end-user needs in their workflow.<\/td>Used as a “multiplier” to see if AI speed is being directed at the right goals.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

I recommend picking out some of these metrics and sticking to them. I personally prefer telemetry-based metrics because they provide more value than filling out a survey. Survey-based metrics should be used sparingly, as they provide more of a temperature reading for an organization.<\/p>\n","protected":false},"excerpt":{"rendered":"

https:\/\/miroslawstaron.github.io\/hallucinations.html#\/5 The shift from manual coding to AI-augmented orchestration is no longer a future – it is a reality. Software engineers adopt AI increasingly often and increasingly deep. However, as organizations pour investment into Generative AI tools, a critical question remains: How do we measure the true return on investment? I asked Gemini to analyze … Continue reading “The Synthetic Engineer: Measuring the Real Impact of AI on Software Delivery”<\/span><\/a><\/p>\n","protected":false},"author":68,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"_links":{"self":[{"href":"https:\/\/metrics.blogg.gu.se\/index.php?rest_route=\/wp\/v2\/posts\/1020"}],"collection":[{"href":"https:\/\/metrics.blogg.gu.se\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/metrics.blogg.gu.se\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/metrics.blogg.gu.se\/index.php?rest_route=\/wp\/v2\/users\/68"}],"replies":[{"embeddable":true,"href":"https:\/\/metrics.blogg.gu.se\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1020"}],"version-history":[{"count":1,"href":"https:\/\/metrics.blogg.gu.se\/index.php?rest_route=\/wp\/v2\/posts\/1020\/revisions"}],"predecessor-version":[{"id":1022,"href":"https:\/\/metrics.blogg.gu.se\/index.php?rest_route=\/wp\/v2\/posts\/1020\/revisions\/1022"}],"wp:attachment":[{"href":"https:\/\/metrics.blogg.gu.se\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1020"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/metrics.blogg.gu.se\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1020"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/metrics.blogg.gu.se\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1020"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}