Atlas [OpenAI] and Comet [Perplexity] were able to read the article for two reasons. The first is that, to a website, Atlas’s AI agent is indistinguishable from a person using a standard Chrome browser. . . . Publishers can selectively block certain crawlers using the Robots Exclusion Protocol—and indeed many do. . . .
Furthermore, the MIT Technology Review. . . uses a client-side overlay paywall: the text loads on the page but is hidden behind a pop-up that asks a user to subscribe or log in. While this content is invisible to humans, AI agents like Atlas and Comet can still read it.
| Artificial Intelligence |
| Research Data Curation and Management Works |
| Digital Curation and Digital Preservation Works |
| Open Access Works |
| Digital Scholarship |