Ask an LLM about a well-known company and it answers fast, and usually gets it right. Ask something obscure and it can be just as confident and completely wrong. This episode traces that behavior back 40 years, to two rival ideas about how to make a machine understand the world, and shows how their recent fusion is what actually answers questions about your company in AI Search today.
What you’ll hear in this episode:
- Why hallucinations run on the exact same machinery that produces correct answers
- The hand-built approach: Cyc, triples, the semantic web, and how Freebase became Google’s Knowledge Graph
- The pattern-learning approach, and how Word2Vec turned a word’s meaning into math you can do arithmetic on
- How the 2017 Transformer and “attention” let a model read a whole sentence in context, and how next-word prediction trained GPT-3
- Why RAG bolts the two systems together, and what it means that you can edit the “knower” but only earn the “talker”
Chapter Headings
[00:02:09] Why an LLM can be confidently wrong
[00:05:45] Teaching machines by hand: Cyc and the 1980s
[00:07:18] Triples, the semantic web, and Freebase
[00:08:51] Google's Knowledge Graph: things not strings
[00:14:08] The other approach: patterns, not facts
[00:16:00] Word2Vec turns meaning into math
[00:18:05] Attention and the Transformer
[00:23:07] BERT, GPT-3, and next-word prediction
[00:26:25] Why hallucinations happen
[00:28:44] Bolting the two brains together with RAG
[00:35:42] The takeaway: the knower you edit, the talker you earn
Resources
Cyc Project (1984) — https://www.cycorp.com/ | https://en.wikipedia.org/wiki/Cyc
Freebase (2007–2016) — https://en.wikipedia.org/wiki/Freebas...)
Wikidata — https://www.wikidata.org/ | https://www.wikidata.org/wiki/Wikidat...
Google Knowledge Graph (2012) — https://blog.google/products/search/i... | https://support.google.com/knowledgep...
Word2Vec (2013) — https://arxiv.org/abs/1301.3781
"Attention Is All You Need" (2017) — https://arxiv.org/abs/1706.03762
BERT (2018) — https://arxiv.org/abs/1810.04805
Retrieval Augmented Generation (2020) — https://arxiv.org/abs/2005.11401
GPT-3 (2020) — https://openai.com/api/
Find the transcript for this episode and additional resources on our blog: https://victorious.com/podcast/history-of-ai-search/