Devin.ai is an impressive generative AI solution that claims to be a new kind of AI engineer designed to perform software engineering tasks autonomously. Cognition Labs, Devin’s creator, said Devin is capable of responding to text commands and can be assigned tasks such as assessing the benchmark performance of an application. It will then build a plan and configure the required tools using its own command line interface (CLI), code editor and browser through which it can access, read and, for example, comprehend documentation using a reasoning engine and long-term planning capability based on advances in reinforcement learning.

Those capabilities make it possible for Devin to, for example, build a website, autonomously identify and fix bugs in codebases, deploy applications and even train other AI models. But there are a number of questions yet to be answered before Devin is ready for wider adoption.

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

For instance, whether or not every question from an organization will be used to train Devin.ai remains to be determined. Many experts believe that some large language model implementations, with almost 2 trillion parameters, cannot be comprehended even by their creators. There are 86 billion neurons in the human brain. Now, there’s not a one-to-one comparison to the human brain, but the point is they are both complex systems. GPT-4’s neural network contains 1.7 trillion parameters. The main point is that, regardless of what anyone says about their LLM implementation, it’s reasonable to assume they don’t know how it works internally. OpenAI states in their license agreement that they do not train on client data, but it is noteworthy that they didn’t have this policy early on.

Thoughts on Devin.ai

One of Devin’s most intriguing aspects is its autonomous operation through its shell code editor and web browser. Devin AI claims to set a new benchmark in software coding, boasting the ability to pass practical engineering interviews and complete real jobs on platforms like Upwork. However, as a software engineer with 45 years of experience, I take these claims with a hefty chaser of skepticism. Devin.ai may introduce many security concerns, especially when dealing with sensitive information or performing actions on services, databases and APIs, as well as the handling of sensitive information and the potential for unintended actions. This emphasizes the importance of developing robust security measures to mitigate these risks introduced by tools like Devin.ai. While I’m not against AI, I am cautious about Devin’s potential to replace human full-stack software engineers. 

Back in the Day

One of the biggest obstacles in selling hosted Chef to regulated companies in the early days was the concern that ten young kids out of Seattle would have access to systems that create billions of dollars worth of infrastructure. Are these same institutions going to trust ten kids out of Silicon Valley with Devin, which may present similar risks? 

Let’s set aside the infrastructure issues for a moment. What are your opinions on the robotic process automation (RPA) revolution that has taken place in the last decade? For one, consider that it has caused a technical debt nightmare. RPAs create many one-off technical problems that can be difficult to solve. One of the great things about the DevOps movement was that it formed a bottleneck between novel ideation and managed scale while consistently proving speed and flow. I’m a huge fan of development tools like Microsoft’s Copilot for development and operations. However, like RPA, I’m deeply concerned about a tool that does both. Consider the implications of Devin.ai-generated solutions and their impact on secure software supply chains and automated governance. Organizations are spending billions of dollars trying to figure this out, and we are still in very early days as far as coming up with solutions.

Conclusion

Devin.ai is a fascinating generative AI tool that is generating a lot of well-deserved buzz in the industry. However, I recommend large organizations be cautious when using it. While monitoring ChatGPT is always a good practice in highly-regulated organizations, I suggest being extra careful when using a tool like Devin.ai. Following Devin.ai’s ideas and progress is a great idea; however, until or unless they create a bring-your-own model implementation, I would proceed cautiously. Organizations should look at useful examples to help understand how you might build your own “Devin.” Here are some resources to check out:

• Matt Wallace’s LinkedIn Post
• Opendevin
• Your Devin

Devin represents a significant leap forward in the realm of AI and software development, and it is a stepping stone toward a future where AI can act as a true collaborator or co-worker rather than a replacement for human engineers. I look forward to advancements that will allow generative AI to tackle more complex, integrated systems and contribute meaningfully to the software development process.