The data set that Eleuther is using is more diverse than GPT-3, and it avoids some sources such as Reddit that are more likely to include dubious material. This is one reason that OpenAI has given for not releasing the full version of GPT-3. It’s also conceivable that a tool like GPT-3 could be used to generate fake news or fraudulent messages. Because GPT-3 and similar large language models draw from random text, they can reproduce bias or produce abusive or discriminatory speech. The project highlights another challenge with opening access to powerful AI systems. “We are definitely thankful that they aggregated all this data into one resource,” Iyyer says. Iyyer says this might help produce a program with a more subtle grasp of language. This includes training an algorithm to predict which sections of a book such as Jane Eyre will be cited in a particular piece of criticism.

Mohit Iyyer, a computer science professor at the University of Massachusetts Amherst, is using data and models from Eleuther to mine literary criticism for insights on famous texts, among other projects. Besides releasing powerful language algorithms modeled after GPT-3, he says the Eleuther team has curated and released a high-quality text data set known as the Pile for training NLP algorithms. Rush, who isn’t affiliated with Eleuther, says the project is one of the most impressive of a growing number of open source efforts in NLP. The code for GPT-3 has not been released, but the few dozen researchers behind Eleuther, who come from across academia and industry, are drawing on papers that describe how it works.

Sentence generator algorithm