AI Tools for Vulnerability Reports Criticized by Developer
Daniel Steenberg, author of the network utility Curl, has expressed criticism towards the use of artificial intelligence (AI) tools in creating vulnerability reports. While these reports may appear high-quality and contain detailed information presented in a normal language, they often lack thoughtful analysis and can be misleading, replacing real problems with seemingly well-crafted but ultimately useless content.
The project known as curl offers rewards for identifying new vulnerabilities and has received a total of 415 reports on potential issues. However, only 64 of these reports have been confirmed as actual vulnerabilities, while 77 were determined to be unrelated to security errors. This means that a significant 66% of all reports provided no useful information and instead wasted developers’ time, which could have been spent on more productive tasks.
Developers are burdened with the task of analyzing these useless reports and double-checking the information multiple times. The reports’ external quality creates a false sense of credibility that the developer might have missed something important. On the other hand, creating such a report requires minimal effort from the person submitting it, as they simply rely on AI helpers without verifying the existence of an actual problem. This reliance on luck in the hope of receiving compensation undermines the effectiveness of the project.
Two examples of these misleading reports are provided. One report, sent on HackerOne, claimed that a patch for a dangerous vulnerability (CVE-2023-38545) was publicly available before the planned disclosure. However, upon analysis, it was discovered that the report contained a mix of information about unrelated issues and excerpts from past vulnerabilities. The information appeared new and relevant but lacked any connection to reality.
The second example involves a message sent by a user on December 28, reporting a buffer overflow in the websocket handler. Upon investigation, it was found that the report lacked accurate details and failed to highlight the actual problem.