A study of 800 software developers working on large engineering teams that have adopted the GitHub Copilot generative artificial intelligence (AI) tool finds limited gains in productivity are being achieved.

Conducted by Uplevel Data Labs, a provider of a software engineering intelligence platform, the study of a test and control group published today also notes that the number of vulnerabilities being created by developers using GitHub Copilot increased.

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

Specifically, developers using GitHub Copilot have not seen any improvement or decrease in pull request (PR) cycle time or overall throughput in terms of the number of pull requests merged.

Developers, however, have experienced a 41% increase in bugs within pull requests.

Matt Hoffman, a product manager and data analyst at Uplevel, said those results suggest that overall code quality is being adversely impacted by an AI tool that was trained using code of varying quality that OpenAI collected from across the Web. GitHub Copilot is based on a large language model (LLM) originally developed by OpenAI that GitHub licenses through its parent company Microsoft.

At the same time, the study suggests that burnout as measured by the amount of time developers spend working outside of standard hours is decreasing. However, that reduction isn’t attributable to generative AI because both the control and test groups saw a decline in the amount of time spent working outside of standard hours, with those that did not have access to GitHub Copilot seeing a 28% reduction compared to a 17% decline for developers that did have access.

It’s still early days when it comes to using generative AI tools to write code. The first generation of these tools is steadily going to be replaced by another wave of these tools that have been trained using code that has been vetted for quality. In addition, those tools will also take advantage of more advanced reasoning engines and AI agents to automate workflows.

In the meantime, however, it’s clear that human developers need to scrutinize the code being generated by the first generation of these tools closely. In addition to injecting more vulnerabilities into code, the code being generated may not run as efficiently as code created by a professional developer, resulting in increased costs.

Of course, generative AI tools also make coding more accessible to a wider range of developers who previously may have stopped coding simply because the effort required was too tedious. Overall, the amount of code being created by developers has increased. It’s just not all that code has impacted PR cycle times for better or worse, according to the Uplevel study. It does, however, remain to be seen if an increased number of vulnerabilities in the code created does lead to an increase in application security incidents.

Ultimately, each organization and the developers that work for them will need to determine how much to rely on generative AI tools. After all, it’s not the machines that created code used in an application that will be held accountable for its overall quality but rather the DevOps teams that allowed it to become incorporated into the software build in the first place.