A recent study showed that program assistants based on artificial intelligence can make more errors to the code than to help in writing it. Technologies based on generative AI promised to increase the efficiency of programmers, but dry numbers talk about something else.
Uplevel, specializing in code management, conducted a study of the effectiveness of programmers using a popular tool from GitHub – Copilot. The results of the experiment turned out to be unexpected: the use of Copilot did not lead to a significant increase in productivity, but on the contrary, caused an increase in code errors by 41%.
For the study, Uplevel has monitored the work of 800 developers for three months before the use of Copilot and for three months after its implementation. The performance assessment was based on the time of adding code to the repository (poull-recruitment) and the total number of executed requests.
The results showed that “Copilot did not affect the speed of the developers, did not accelerate the coding process and did not improve the quality of the code.” Contrary to the expectations of the Uplevel team, the temporary costs of pool-records did not decrease, and the defectiveness of the code did not decrease, as was originally assumed.
One of the reasons for the low efficiency of Copilot may be that it is based on large language models (LLM), which are prone to so -called “hallucinations” – generating incorrect information and creating incorrect code. A study conducted by scientists from Texas University in San Antonio revealed that LLM often offers “non-existent” files or libraries.
Anxiety regarding the quality of the code generated by artificial intelligence is also shared by representatives of the industry. GEHTSOFT CEO Ivan Gekht noted that it is so difficult to understand and correct errors in the code created by AI that it’s easier to write it from scratch.
Thus, despite the expectations of AI instruments in encoding, they do not yet justify the declared advantages, sometimes even complicating the work of programmers.