Discord's Safety System Snares Innocents A bug in Discord's content filtering system led to thousands of accounts being incorrectly banned due to posts containing square grid images mistakenly identified as child sexual abuse material.
The affected users were locked out for over two months, highlighting the inherent risks of outsourcing judgment calls to algorithms.
The incident underscores the limitations of AI powered moderation tools, which rely on pattern recognition to identify suspicious content but can be easily fooled by innocent uploads.