Generative AI is a subset of artificial intelligence designed to create and produce new content using algorithms to generate images, text, and music. Generative AI models like ChatGPT replicate the structure and style using patterns from data they have been trained on to generate responses. Generative AI can be used in a myriad of ways from creative content generation to problem-solving tasks.
John Warner, author of Why They Can't Write and The Biblioracle Recommends, summarizes ChatGPT in the following way: “It’s important to understand what ChatGPT is, as well as what it can do. ChatGPT is a Large Language Model (LLM) that is trained on a set of data to respond to questions in natural language. The algorithm does not “know” anything. All it can do is assemble patterns according to other patterns it has seen when prompted by a request. It is not programmed with the rules of grammar. It does not sort, or evaluate the content. It does not “read”; it does not write.” (Warner, 2022)
NYU Library' Machines and Society guide has a fantastic overview of Large Language Models that we highly recommend reviewing in full. This excerpt cover the basics:
Large Language Models (LLMs) refer to large general-purpose language models that can be pre-trained and then fine-tuned for specific purposes. They are trained to solve common language problems, such as text classification, question answering, document summarization, and text generation. The models can then be adapted to solve specific problems in different fields using a relatively small size of field datasets via fine-tuning.
The ability of LLMs taking the knowledge learnt from one task and applying it to another task is enabled by transfer learning. Pre-training is the dominant approach to transfer learning in deep learning.
LLMs predict the probabilities of next word (token), given an input string of text, based on the language in the training data. Besides, instruction tuned language models predict a response to the instructions given in the input. These instructions can be "summarize a text", "generate a poem in the style of X", or "give a list of keywords based on semantic similarity for X".
This 15 minute video from Google Cloud Tech provides a more thorough introduction to Large Language Models.
One of the greatest challenges of AI is that tools are arriving and evolving so quickly. The links below offer a few collections of AI tools and resources.
The Guardians' How AI Chatbots like ChatGPT and Bard work is a great visual walkthrough of how AI Chatbots function and operate.
NYU Library's Machines and Society guide has an excellent overview of ChatGPT, how it works, and how to use it. This excerpt is from the introduction to this guide:
In essence, ChatGPT is a chatbot interface to a series of models that power it. It is capable of generating natural language and code in a dialogue format for a variety of tasks. ChatGPT was released in November 2022 by the company OpenAI.
ChatGPT models
The newest model is GPT-4, a large multimodal model released in March 2023. Compared with its predecessor GPT-3.5, GPT-4 can process more than one modality of information, which accepts image and text inputs and produces text outputs. Read more in GPT-4 Technical Report (PDF).
GPT-4 also has an expanded memory, or the limit to how much the models can "remember" in a conversation with the user. This limit includes the token count from both the prompt and completion. Specifically, GPT-4 can process about 32,000 tokens in a query. By comparison, GPT-3.5-turbo can process around 4,000 tokens.
Datasets used to train GPT-3, as documented in its technical report (PDF), include Common Crawl, WebText2, Books1, Books2, and Wikipedia, although sources of Book1 and Books2 are not entirely transparent.
One important consideration when using ChatGPT (whether version 3.5 or 4) is, at the time of this writing (21-Aug-2023), that the current knowledge cutoff date is September 2021. This means that the vast majority of the text used in training the large language model is from before this date, and the model therefore has very little information to draw upon when discussing more recent events.
This three hour talk, with questions, by Stephen Wolfram (Founder of Wolfram/Mathematica) was originally streamed on Twitch on February 17, 2023. The talk begins starts at 9:53.
Tucker, E. (2022, March 8). Artifice and Intelligence. Center on Privacy & Technology at Georgetown Law. https://medium.com/center-on-privacy-technology/artifice-and-intelligence%C2%B9-f00da128d3cd
Furze, L. (2023, January 17). ChatGPT in Education: Back to Basics. Leon Furze. https://leonfurze.com/2023/01/17/chatgpt-in-education-back-to-basics/
Chrisinger, B. (2023, February 22). Opinion | It’s Not Just Our Students—ChatGPT Is Coming for Faculty Writing. The Chronicle of Higher Education. https://www.chronicle.com/article/its-not-just-our-students-ai-is-coming-for-faculty-writing
McMurtrie, B. (2022, December 13). Teaching Experts Are Worried About ChatGPT, but Not for the Reasons You Think. The Chronicle of Higher Education. https://www.chronicle.com/article/ai-and-the-future-of-undergraduate-writing
Vilone, G., & Longo, L. (2020). Explainable Artificial Intelligence: A Systematic Review (arXiv:2006.00093). arXiv. https://doi.org/10.48550/arXiv.2006.00093
Wolfram, S. (2023, February 14). What Is ChatGPT Doing … and Why Does It Work?.Stephen Wolfram. https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/