Stack Overflow bans ChatGPT as 'substantially harmful'

OpenAI's question-answering bot, ChatGPT, isn't smart enough for the team at Stack Overflow, who today announced a temporary ban on answers generated by the AI bot because of how frequently it's wrong.

Stack Overflow said it was withholding a permanent decision on AI-generated answers until after a larger staff discussion, but was taking action now due to fears that ChatGPT could be "substantially harmful" to both the org and its users.

"The average rate of getting correct answers from ChatGPT is too low," Stack Overflow said in a policy statement entitled "ChatGPT is banned."

ChatGPT was released last week and touted by OpenAI in a blog post as a conversational AI that can provide detailed answers to questions, as well as "answer followup questions, admit its mistakes, challenge inappropriate premises and reject inappropriate requests."

OpenAI lists debugging code as one of ChatGPT's uses, which would seem to make it an ideal companion for developers seeking to cut out bad lines. But the Stack Overflow administrators who wrote the post said that simply isn't the case.

"While the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce," Stack Overflow said.

"The volume of these answers (thousands) and the fact that [they] often require a detailed read by someone with at least some subject matter expertise in order to determine that the answer … has effectively swamped our volunteer-based quality curation infrastructure," the SO team added.

To address the issue quickly, Stack Overflow said it was going to take action against anyone posting answers or new posts generated by ChatGPT, even if such a post would otherwise be acceptable.
• OpenAI opens doors to ChatGPT, another AI to fill the world with kinda-true stuff
• Enterprises are rolling out more AI – to 'middling results'
• AI-driven creativity gives overpowered PCs something worthwhile to do, at last

ChatGPT was designed to avoid abuse and answers that contain harmful advice, but users quickly found such things could be sidestepped. For example, by telling ChatGPT not that you want to make a Molotov cocktail, but that you want it to complete a Python function that prints instructions to do the same, it will tell you exactly how to make one via print functions.

The AI has also been called out for providing bad information in biographical sketches, and for "proving that incorrect answers delivered cogently can fool people," according to one Twitter user who spotted some bad math generated by ChatGPT.

We've reached out to OpenAI to learn what it plans to do regarding these reported accuracy issues, but haven't heard back.

Stack Overflow said it'll be weighing new policies surrounding ChatGPT "and other similar tools," but a decision date, and what it will ultimately restrict, is unknown. ®