GitHub Copilot Tops AI Code Assistant Research Report – Visual Studio Magazine

News

GitHub Copilot leads research report on AI code assistants

GitHub led research firm Gartner's first Magic Quadrant report of AI code assistant vendors, leading in completeness of vision and ability to execute.

This is perhaps not surprising, as GitHub Copilot was the first AI coding assistant to break free from the constraints of machine learning about three years ago and helped usher in the GenAI era with its deep learning and natural language processing (NLP) capabilities.

Gartner now describes the market as follows:

Gartner defines AI code assistants as tools that help generate and analyze software code and configuration. The assistants use base models such as large language models (LLMs) that are optionally optimized for code, or program understanding technologies, or a combination of both. Software developers call on the code assistants to generate, analyze, debug, repair, and refactor code, create documentation, and translate code between languages. Code assistants integrate with developer tools such as code editors, command-line terminals, and chat interfaces. Some can be customized to an organization's specific code base and documentation.

In general, these tools have evolved rapidly since Copilot's debut, moving beyond initial, simple code completion suggestions. According to Gartner, AI code assistants now work in a range of use cases.

• Code generation: Developers use the code editors in AI code assistants to auto-complete code and generate functions, helping them complete programming tasks faster.
• Code debugging: Developers use AI code assistants to detect and fix errors in code, allowing them to fix bugs without having to ask colleagues or search the internet for solutions.
• Code modernization: Developers use AI code assistants to understand complex dependencies between many programs, helping them reduce technical debt and modernize code.
• Creating and testing artifacts: Developers use AI code assistants to generate acceptance tests (for example in Gherkin format) and unit tests from user stories.
• Code explanation: Developers use AI code assistants to get explanations of code in natural language, helping them understand complex and unfamiliar code.

The very first report highlighted GitHub Copilot's AI-powered code suggestions and contextual support, saying, “The company's operations are geographically diversified and its customers tend to be large organizations from various sectors. GitHub makes GitHub Copilot available free of charge to active maintainers in the open source community, as well as teachers and students. GitHub extends Copilot with features such as Copilot Workspace for a collaborative AI-native developer environment, Copilot Extensions for seamless tool integration, and enhanced security and compliance.”

In addition to GitHub Copilot, Google Cloud, Amazon Web Services (AWS) and GitLab are also represented in the Leaders quadrant.

Mandatory features listed by Gartner for this emerging market include:

• Code completion from natural language (e.g. comment).
• Multi-line code completion with center fill and the ability to include integrations for multiple code editors.
• Ability to use the Code Wizard in more than one vendor ecosystem.
• Guarantee that base models are not trained using customer code or documentation (except for approved fine-tuning).
• Conversational chat interface integrated into the development environment.

Other common features include local or private cloud instances and filters for biased code, explicit language, and images.

“The vision behind GitHub Copilot is simple: to augment the innate human creativity of every developer through generative AI,” Microsoft-owned GitHub said in a blog post last week announcing the report. “Our goal was never to build technology for technology's sake, but to increase the happiness and productivity of every developer by keeping them in the flow longer, helping them move faster, and lowering the overall barrier to entry. With millions of developers and over 77,000 organizations using Copilot, we feel like we're rapidly moving toward that goal and realizing our vision.”

Considering that two of the top four are cloud giants and GitHub is owned by a cloud giant, GitLab, which offers an AI-powered DevSecOps platform, was the only leader that is not a cloud giant.

“AI code assistants go beyond mere code generation and completion,” GitLab said in its own post acknowledging the report. “They are collaborative partners that increase developer efficiency by improving code quality and continuously learning. By automating routine tasks and providing intelligent suggestions, assistants like GitLab Duo – our suite of AI-powered features – free up developers to focus on solving higher-level problems.”

While Duo is GitLab's offering in this space, Google was evaluated for its Gemini Code Assist tool, while AWS was evaluated for its Amazon Q Developer, formerly called Codewhisperer.

In addition to evaluating vendors, the report also lists some strategic planning assumptions that provide insight into Gartner's market outlook:

• By 2027, the number of platform development teams using AI to improve every phase of the software development lifecycle (SDLC) will increase from 5 percent to 40 percent.
• By 2027, 80 percent of companies will have integrated AI-powered testing tools into their software engineering toolchain, a significant increase from about 15 percent in early 2023.
• By 2027, 25 percent of software defects entering production will be due to a lack of human control over AI-generated code. This is a significant increase compared to 2023, when this figure was less than one percent.
• By 2028, 90 percent of enterprise software engineers will use AI code assistants, up from less than 14 percent in early 2024.
• By 2028, the use of generative AI (GenAI) will reduce the cost of modernizing legacy applications by 30 percent compared to 2023 levels.

While Gartner typically charges for its reports, this and many other Magic Quadrant reports are available free of charge from evaluated vendors who have permission to provide licensed versions for distribution, which can be found with a simple Internet search.