Study: ChatGPT’s programmer’s help answers have errors 52% of the time

Source: Arxiv

Over the last decade, Q&A platforms have played a crucial role in how programmers seek help online. The emergence of ChatGPT, however, is causing a shift in this pattern. Despite ChatGPT’s popularity, there hasn’t been a thorough investigation into the quality and usability of its responses to software engineering queries. To address this gap, we undertook a comprehensive analysis of ChatGPT’s replies to 517 questions from Stack Overflow (SO). We assessed the correctness, consistency, comprehensiveness, and conciseness of these responses. Additionally, we conducted an extensive linguistic analysis and a user study to gain insights into the linguistic and human aspects of ChatGPT’s answers. Our examination revealed that 52% of ChatGPT’s answers contain inaccuracies

Continua qui: [2308.02312] Who Answers It Better? An In-Depth Analysis of ChatGPT and Stack Overflow Answers to Software Engineering Questions

If you like this post, please consider sharing it.

Leave a Comment

Your email address will not be published. Required fields are marked *