Further, we want to know: Is it human, or AI? Is that picture real? That song? That video? Publishing platform Substack is trying to add some clarity by partnering with AI detector Pangram to help readers understand how much AI is in all the content they consume in the publishing space.
The tool is available to all Substack subscribers and applies to posts, notes, comments and replies published on or after Tuesday that are longer than 100 words. Creators will also be able to scan their drafts with Pangram to see how much AI has been discovered and can even provide a statement to explain their writing process, Substack said.
Web and iOS users can try Pangram’s features now, and Android users will get access later.
In a blog post announcing the new feature, Substack founder and CEO Chris Best said his platform aims to avoid “Claudefishing,” which is when a reader consumes content without realizing it was generated by AI.
“When readers have to question whether what they’re reading is true, it undermines trust in writing and threatens the lives of writers — including those who thoughtfully use AI tools to produce work they believe in,” Best wrote.
From CNET: AI Slop Is Ruining the Internet. These are the people who are fighting to save it
Pangram estimates that about 40% of the text on some social networks is AI-gene.limited, with a good two-thirds of that on the LinkedIn database. And AI growth company Graphite recently published a white paper that says AI is now writing as many online articles as humans. A Gartner survey found that 68% of people question whether the content they see is authentic, and that half of consumers prefer brands that don’t use AI in their marketing.
I check it for myself
I just joined Substack, but I wanted to check out the new Pangram AI discovery tool. The tool works on posts, notes, comments and replies, so I decided to use posts as my temporary test lab. First, I logged into my Substack account and published a new post. I copied and pasted the text from the article I had written – by myself, no AI – into the Substack post and published it.
A few minutes later, I clicked live post. In the three-dot menu at the top right of the page, I saw the option “Scan AI text”. I clicked on it, and a small window appeared that accurately stated that my title was “Fully Written by Humans.” A line appears below this showing the percentage of text that is AI, AI-assisted, or human.
Next I took the same post but added a long, fully developed AI section that I attached to ChatGPT. When I did “Scan text for AI,” the tool reported that 10% of the article was written by AI, which is an almost accurate estimate.
Next I published a fully AI generated post, and the scan was correctly identified as 100% AI generated.
In another test, I added a few AI-generated sentences to a post that was originally human, and the scan marked it as 100% human-generated. Close, but it’s wrong. So, the Pangram tool worked very well in my sloppy testing.
Although I was satisfied with how well Substack’s AI detection system worked on my content, I was disappointed to learn that one of my favorite bloggers apparently went full AI on his latest post. Some of the authors I examined, however, used entirely humanized text.
What about misconceptions?
In his announcement, Best acknowledged that the AI detection tool is “not perfect,” but cited an independent study that found Pangram’s software was 97.5% accurate at detecting fully generated AI text. The University of Maryland found that Pangram was 99.3% accurate at identifying “human AI text,” meaning that a human rewrote or paraphrased the AI’s content.
Non-AI substack writers are understandably concerned about being mislabeled as AI-assisted writers. As a result, the risk of false positives is an important issue. Substack said creators can scan their content to see if it will be labeled as containing AI while it’s still in draft form. If Pangram incorrectly identifies the AI text, the creator can report the error to Substack and issue a false scan.
To promote greater transparency, publishers can now include a “How I do this” statement if they want readers to know how much, if any, AI they have used in their content. Best of all, he said, Pangram can’t tell if the AI was used to gather information or as a source of written pieces.
“We are not against people using AI to help their work, and we think people should be free to choose what tools they use to express themselves,” said Best. “But people should know what they’re getting.”
See the full history
