{ "cells": [ { "cell_type": "markdown", "id": "6da76396", "metadata": {}, "source": [ "### An application to generate epigrams\n", "\n", "For our next assignment we are going to pretend that we have been hired by a company that is working on a blogging platform. One of the features that they would like to add is an automated epigram generator. An epigram is a short quote that many authors attach to works like blog posts or chapters in a book that is meant to introduce the content in the work." ] }, { "cell_type": "markdown", "id": "2815bd70", "metadata": {}, "source": [ "One way we could solve this problem would be to use a large LLM. Since large LLMs have a lot of knowledge about the world, they know a large number of quotes and can find a quote to match a block of text. For example, if I ask the larger LLM we have access to produce an epigram for this text:\n", "\n", ">Americans are reading less, sleeping less and partying less. We have fewer marriages, fewer children and fewer friends than we used to. Our children are doing worse in school.\n", ">\n", ">These are complicated phenomena on some level, but on another level it’s pretty simple: Smartphones, social media and the internet are transforming our lives and our culture in ways that are not all for the best. Streaming video is ubiquitous, and some of the smartest people in the world have developed powerful recommendation engines to ensure that we always have an unlimited supply of customized content. Technological progress has, in effect, supercharged television, and as a result people are passively watching more and actively doing less.\n", ">\n", ">Yet rather than saying straightforwardly that this is an unhealthy trend and the US should enlist public policy to reverse it, many critics are focusing on a bank-shot idea: This is a competition policy problem related to the bigness of the largest technology companies. Others have gone full anti-capitalist and have slogans about the evils of tech billionaires.\n", ">\n", ">None of this makes very much sense. Some of the biggest culprits for transforming the information superhighway into a giant time suck are companies such as TikTok and X, which are not particularly large. Some of the biggest technology companies, such as Nvidia, Oracle and Broadcom, are only incidentally involved in the content glut.\n", ">\n", ">More to the point, the issue here is not the classic antitrust problem of a monopolist restricting output to raise prices. There is too much content, and the content is too cheap — in fact, it’s generally free. Tech skeptics who acknowledge this point have, like Lina Khan in her star-making law review article “Amazon’s Antitrust Paradox,” tended to retreat to convoluted theories about corporate power that manifests in non-price ways.\n", ">\n", ">There is a better way to think about the issue of overconsumption of digital content: It is a vice.\n", ">\n", ">The US has a long tradition of public policies designed to reduce the availability of vice products, from alcohol and tobacco to pornography. These industries are regulated not to create a better value proposition for consumers, but to make it worse. What motivates the regulation is not hostility to market capitalism or entrepreneurs, but respect for the power of free markets to create abundance, even of vice products.\n", ">\n", ">Granted, the US has stepped away from vice regulation in recent years. More than half of Americans live in states where marijuana is legal, and there is easy nationwide access to online sports gambling. At the same time, there has been a huge reduction in smoking in the last few decades, a reminder that public policy can constrain the consumption of undesirable goods.\n", "\n", "It will respond with \n", "\n", ">“A wealth of information creates a poverty of attention.” — Herbert A. Simon" ] }, { "cell_type": "markdown", "id": "72dce4c3", "metadata": {}, "source": [ "The problem with using large LLMs is that they are expensive. To bypass this expense we can use a combination of a RAG system and a smaller, cheaper LLM. The smaller LLM doesn't need to know any quotes itself, it just needs to be able to perform simple tasks such as labeling, summarization, and evaluation of potential quotes. A further benefit of using a RAG system is that we can control the pool of potential quotes, which is something that we can't do when we simply ask an LLM to come up with a quote itself." ] }, { "cell_type": "markdown", "id": "573ae56c", "metadata": {}, "source": [ "The first thing we are going to need is a database of quotations.\n", "\n", "We can download one such database from Hugging Face." ] }, { "cell_type": "code", "execution_count": null, "id": "7c72f893", "metadata": {}, "outputs": [], "source": [ "!pip install pandas" ] }, { "cell_type": "code", "execution_count": 2, "id": "c1b71876", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "df = pd.read_json(\"hf://datasets/Abirate/english_quotes/quotes.jsonl\", lines=True)" ] }, { "cell_type": "code", "execution_count": 6, "id": "590b1fb2", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
quoteauthortags
0“Be yourself; everyone else is already taken.”Oscar Wilde[be-yourself, gilbert-perreira, honesty, inspi...
1“I'm selfish, impatient and a little insecure....Marilyn Monroe[best, life, love, mistakes, out-of-control, t...
2“Two things are infinite: the universe and hum...Albert Einstein[human-nature, humor, infinity, philosophy, sc...
3“So many books, so little time.”Frank Zappa[books, humor]
4“A room without books is like a body without a...Marcus Tullius Cicero[books, simile, soul]
............
2503“Morality is simply the attitude we adopt towa...Oscar Wilde,[morality, philosophy]
2504“Don't aim at success. The more you aim at it ...Viktor E. Frankl,[happiness, success]
2505“In life, finding a voice is speaking and livi...John Grisham[inspirational-life]
2506“Winter is the time for comfort, for good food...Edith Sitwell[comfort, home, winter]
2507“Silence is so freaking loud”Sarah Dessen,[just-listen, loud, owen, sara-dessen, silence]
\n", "

2508 rows × 3 columns

\n", "
" ], "text/plain": [ " quote \\\n", "0 “Be yourself; everyone else is already taken.” \n", "1 “I'm selfish, impatient and a little insecure.... \n", "2 “Two things are infinite: the universe and hum... \n", "3 “So many books, so little time.” \n", "4 “A room without books is like a body without a... \n", "... ... \n", "2503 “Morality is simply the attitude we adopt towa... \n", "2504 “Don't aim at success. The more you aim at it ... \n", "2505 “In life, finding a voice is speaking and livi... \n", "2506 “Winter is the time for comfort, for good food... \n", "2507 “Silence is so freaking loud” \n", "\n", " author tags \n", "0 Oscar Wilde [be-yourself, gilbert-perreira, honesty, inspi... \n", "1 Marilyn Monroe [best, life, love, mistakes, out-of-control, t... \n", "2 Albert Einstein [human-nature, humor, infinity, philosophy, sc... \n", "3 Frank Zappa [books, humor] \n", "4 Marcus Tullius Cicero [books, simile, soul] \n", "... ... ... \n", "2503 Oscar Wilde, [morality, philosophy] \n", "2504 Viktor E. Frankl, [happiness, success] \n", "2505 John Grisham [inspirational-life] \n", "2506 Edith Sitwell [comfort, home, winter] \n", "2507 Sarah Dessen, [just-listen, loud, owen, sara-dessen, silence] \n", "\n", "[2508 rows x 3 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df" ] }, { "cell_type": "markdown", "id": "6778f9d8", "metadata": {}, "source": [ "This data set contains 2508 quotations. Each quotation has an associated author and a list of tags that have been applied. The tags make it easier to find quotes by topic.\n", "\n", "Here is some code to isolate the full list of tags." ] }, { "cell_type": "code", "execution_count": 17, "id": "78d64609", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'romantic',\n", " 'unbreakable-vow',\n", " 'speak',\n", " 'mystery',\n", " 'feelings',\n", " 'hazel',\n", " 'hobbit',\n", " 'friendship',\n", " 'vorkosigan',\n", " 'odair',\n", " 'be-yourself-inspirational',\n", " 'taste',\n", " 'gaiman',\n", " 'life-humor',\n", " 'jem-carstairs',\n", " 'misattributed-to-rosa-parks',\n", " 'improvement',\n", " 'won-t',\n", " 'good-reads',\n", " 'nerdfighter',\n", " 'dystopia',\n", " 'city-of-lost-souls',\n", " 'renewal',\n", " 'louisa-may-alcott',\n", " 'advice-for-daily-living',\n", " 'arts',\n", " 'hidden-things',\n", " 'jace',\n", " 'tough-times',\n", " 'david',\n", " 'start-over',\n", " 'minister',\n", " 'christina',\n", " 'hate',\n", " 'inspiration',\n", " 'discipline',\n", " 'brain',\n", " 'will-s-letter-to-tessa',\n", " 'misattributed-steve-jobs',\n", " 'insults',\n", " 'misattributed-marilyn-monroe',\n", " 'wormtail',\n", " 'facts',\n", " 'worry',\n", " 'self-deprecation',\n", " 'sexuality',\n", " 'musical',\n", " 'breaking-up',\n", " 'summer',\n", " 'phenomenal-woman',\n", " 'atticus-finch',\n", " 'pretend',\n", " 'habits',\n", " 'automobile',\n", " 'why',\n", " 'soul',\n", " 'desire',\n", " 'light',\n", " 'magnus-bane',\n", " 'the-perks-of-being-a-wallflower',\n", " 'inspirational-faith',\n", " 'believe-in-yourself',\n", " 'jordan-baker',\n", " '2012',\n", " 'innocence',\n", " 'danger',\n", " 'richness',\n", " 'age-adult',\n", " 'hidden',\n", " 'education-knowledge',\n", " 'unity',\n", " 'dreaming',\n", " 'think-different',\n", " 'principles',\n", " 'hippy',\n", " 'dr-seuss',\n", " 'mottos',\n", " 'purpose',\n", " 'tea',\n", " 'dream',\n", " 'touching',\n", " 'wealth',\n", " 'enemies',\n", " 'aloneness',\n", " 'leaving',\n", " 'waking-up',\n", " 'death-of-a-loved-one',\n", " 'inspire',\n", " 'jane-austen',\n", " 'tobias',\n", " 'relaxation',\n", " 'katniss-and-peeta',\n", " 'ponyboy-curtis',\n", " 'the-little-prince',\n", " 'unfairness',\n", " 'relationships',\n", " 'by-anne-sullivan',\n", " 'romance',\n", " 'names',\n", " 'pretty',\n", " 'forgetting',\n", " 'accomplishment',\n", " 'diversity',\n", " 'nightmares',\n", " 'right',\n", " 'security',\n", " 'lost-innocence',\n", " 'golda',\n", " 'happy',\n", " 'greg-behrendt',\n", " 'clock',\n", " 'coffee',\n", " 'femininity',\n", " 'literacy',\n", " 'heartbreak',\n", " 'serenity',\n", " 'ataraxy',\n", " 'awesome',\n", " 'buddhism',\n", " 'patriotism',\n", " 'less-more',\n", " 'share',\n", " 'a-court-of-mist-and-fury',\n", " 'top-8',\n", " 'television',\n", " 'opportunity',\n", " 'person',\n", " 'agree',\n", " 'reading-books',\n", " 'lies',\n", " 'life-dreams',\n", " 'towns',\n", " 'lost-love',\n", " 'isabelle-lightwood',\n", " 'spiritual',\n", " 'thinking',\n", " 'save',\n", " 'defenses',\n", " 'social-norms',\n", " 'flattery',\n", " 'thing',\n", " 'tyson',\n", " 'science',\n", " 'step-out-of-comfort-zone',\n", " 'adventuring',\n", " 'inner-strength',\n", " 'cows',\n", " 'quotation',\n", " 'frolick',\n", " 'listen',\n", " 'expression',\n", " 'misattributed-einstein',\n", " 'four',\n", " 'behind',\n", " 'soon',\n", " 'name-of-the-wind',\n", " '1955',\n", " 'ruby',\n", " 'hurt',\n", " 'force',\n", " 'double-standards',\n", " 'robin-goodfellow',\n", " 'everything',\n", " 'giving-in',\n", " 'misattributed-stephen-hawking',\n", " 'humor',\n", " 'silence',\n", " 'death',\n", " 'teasing',\n", " 'mother',\n", " 'living-life-to-the-fullest',\n", " 'attraction',\n", " 'guys',\n", " 'advocacy',\n", " 'skeptic',\n", " 'winter',\n", " 'feminism',\n", " 'beatrice',\n", " 'misattributed-grace-hopper',\n", " 'introspection',\n", " 'companionship',\n", " 'dionysus',\n", " 'doing',\n", " 'finnick',\n", " 'melancholy',\n", " 'parting',\n", " 'only-way',\n", " 'fuck',\n", " 'stories-writing',\n", " 'seize-the-day',\n", " 'snape',\n", " 'owen',\n", " 'sphinx-without-a-secret',\n", " 'frank',\n", " 'bossypants',\n", " 'science-of-mind',\n", " 'creation',\n", " 'life-lessons',\n", " 'rationalism',\n", " 'the-hunger-games',\n", " 'misattributed',\n", " 'concern',\n", " 'inkheart',\n", " 'honor',\n", " 'personal-responsibility',\n", " 'sausage',\n", " 'fresh-start',\n", " 'clouds',\n", " 'mindful',\n", " 'evidence',\n", " 'book-the-count-of-monte-cristo',\n", " 'your-way',\n", " 'metaphor',\n", " 'sufi',\n", " 'self-image',\n", " 'success',\n", " 'memory',\n", " 'thoughts',\n", " 'haunt',\n", " 'eleven-minutes',\n", " 'acorns',\n", " 'courtroom-drama',\n", " 'valour',\n", " 'glittering-eyes',\n", " 'sweetness',\n", " 'will',\n", " 'prince-charming',\n", " 'giving',\n", " 'responsibility',\n", " 'caring',\n", " 'city-of-bones',\n", " 'goodbye',\n", " 'loner',\n", " 'sad',\n", " 'out-of-control',\n", " 'judge',\n", " 'hunger-games',\n", " 'jail',\n", " 'act',\n", " 'consul',\n", " 'absence',\n", " 'free-will',\n", " 'ferret',\n", " 'sea',\n", " 'secret',\n", " 'sin',\n", " 'missing',\n", " 'man-s',\n", " 'unsourced',\n", " 'silly-chinese-canadian-baby-man',\n", " 'madness',\n", " 'troublemakers',\n", " 'neverland',\n", " 'lookingforalaska',\n", " 'helping-others',\n", " 'human-connections',\n", " 'observational-comedy',\n", " 'make-believe',\n", " 'real-life',\n", " 'blood',\n", " 'sebastian',\n", " 'misattributed-oscar-wilde',\n", " 'enthusiasm',\n", " 'risk',\n", " 'god-religion-happiness',\n", " 'isabelle',\n", " 'sirius-black',\n", " 'nagasawa',\n", " 'physics',\n", " 'parties',\n", " 'wisher',\n", " 'tired',\n", " 'goals-in-life',\n", " 'psychotherapist',\n", " 'qoutes-on-writing',\n", " 'agnosticism',\n", " 'zeus',\n", " 'miracle',\n", " 'seeing',\n", " 'yourself',\n", " 'passion',\n", " 'shock',\n", " 'new-year',\n", " 'experience-mistakes-wisdom',\n", " 'cinna',\n", " 'assurance',\n", " 'safety',\n", " 'infinite-jest',\n", " 'society',\n", " 'fire',\n", " 'self-acceptance',\n", " 'sow',\n", " 'misprints',\n", " 'eternity',\n", " 'rescuing',\n", " 'rythmic',\n", " 'mara-dyer',\n", " 'owen-armstrong',\n", " 'self-deception',\n", " 'sense',\n", " 'ideals',\n", " 'moving-forward',\n", " 'neighborhoods',\n", " 'wrong',\n", " 'congratulations',\n", " 'just-listen',\n", " 'singer',\n", " 'waiting',\n", " 'teenagers',\n", " 'philosphy',\n", " 'just-do-it',\n", " 'strength',\n", " 'principle',\n", " 'eragon',\n", " 'love-friendship',\n", " 'good-books',\n", " 'self-containment',\n", " 'give',\n", " 'viereck-interview',\n", " 'feathers',\n", " 'those-who-matter',\n", " 'darkness',\n", " 'age',\n", " 'enjoy-the-journey',\n", " 'haunting',\n", " 'fresh-starts',\n", " 'humor-heaven',\n", " 'catching-fire',\n", " 'christians',\n", " 'dorian-gray',\n", " 'simple',\n", " 'truth-telling',\n", " 'chinese',\n", " 'painting',\n", " 'loud',\n", " 'homicide',\n", " 'judgment',\n", " 'adoration',\n", " 'troubles',\n", " 'how-many',\n", " 'never-stop-believing',\n", " 'quotes',\n", " 'dreams',\n", " 'dark',\n", " 'talking',\n", " 'misattributed-a-a-milne',\n", " 'old-books',\n", " 'morning',\n", " 'excuse',\n", " 'nonfiction',\n", " 'inspirational',\n", " 'simile',\n", " 'nin',\n", " 'best-friends',\n", " 'molly-weasley-anger',\n", " 'ghost',\n", " 'strength-and-courage',\n", " 'gandalf',\n", " 'home',\n", " 'rena-silverman',\n", " 'wife',\n", " 'vanity',\n", " 'jace-lightwood',\n", " 'humor-tattoo',\n", " 'encouragement',\n", " 'sirius',\n", " 'becca-fitzpatrick',\n", " 'misattributed-eleanor-roosevelt',\n", " 'life-philosophy',\n", " 'learning',\n", " 'vlog',\n", " 'theresa-gray',\n", " 'skepticism',\n", " 'dark-humor',\n", " 'forgiving',\n", " 'americans',\n", " 'atheist',\n", " 'tyler-durden',\n", " 'death-peace',\n", " 'empowerment',\n", " 'self-realization',\n", " 'e-readers',\n", " 'smiles',\n", " 'whiskey',\n", " 'drive-out',\n", " 'humans',\n", " 'blues',\n", " 'peer-pressure',\n", " 'unhappy-marriage',\n", " 'openness',\n", " 'experience',\n", " 'books',\n", " 'shrugged',\n", " 'misattributed-to-mother-teresa',\n", " 'johnny-cade',\n", " 'judging-from-experience',\n", " 'marriage',\n", " 'incrementalism',\n", " 'stories',\n", " 'fiction',\n", " 'obvious',\n", " 'inner-peace',\n", " 'tintern-abbey',\n", " 'intelligent-life',\n", " 'individuality',\n", " 'december',\n", " 'chocolate',\n", " 'disappointment',\n", " 'tfios',\n", " 'santiago',\n", " 'ordinary',\n", " 'failure',\n", " 'judgement',\n", " 'busy',\n", " 'humor-mustache',\n", " 'betrayal',\n", " 'awe',\n", " 'atheism',\n", " 'jordan-kyle',\n", " 'distance',\n", " 'mermaids',\n", " 'heartwarming',\n", " 'hypocrisy',\n", " 'advice-for-writers',\n", " 'loyalty',\n", " 'witch',\n", " 'ruby-anger',\n", " 'rebels',\n", " 'writing-advice',\n", " 'gandhi',\n", " 'commercial',\n", " 'dark-side',\n", " 'doctor-who',\n", " 'telekinetics',\n", " 'anything-can-be',\n", " 'dragons',\n", " 'kissing',\n", " 'deceit',\n", " 'inspirational-quotes',\n", " 'smile',\n", " 'procrastinate',\n", " 'self-determination',\n", " 'confession',\n", " 'jokes',\n", " 'men',\n", " 'being-positive',\n", " 'famous',\n", " 'social',\n", " 'saint-sinner',\n", " 'anais',\n", " 'lemons',\n", " 'hope',\n", " 'lord-henry-wotton',\n", " 'mormont',\n", " 'originality',\n", " 'glass',\n", " 'lesson-on-living',\n", " 'devil',\n", " 'dizziness',\n", " 'education',\n", " 'rulers',\n", " 'attitude',\n", " 'feminist',\n", " 'true-love',\n", " 'deception',\n", " 'literature',\n", " 'girls',\n", " 'fun',\n", " 'mind',\n", " 'lupin',\n", " 'thorns',\n", " 'pets',\n", " 'lust',\n", " 'misanthropy',\n", " 'energy',\n", " 'challenges',\n", " 'pleasure',\n", " 'rewriting-history',\n", " 'gift',\n", " 'paradox',\n", " 'sorcerers-stone',\n", " 'unpredictability',\n", " 'wring',\n", " 'independence',\n", " 'dreamer',\n", " 'be-yourself-quote',\n", " 'toltec-wisdom',\n", " 'self-esteem',\n", " 'equality',\n", " 'extraterrestrials',\n", " 'safran',\n", " 'vegetarianism',\n", " 'nerds',\n", " 'requirements',\n", " 'silliness',\n", " 'simon-lewis',\n", " 'bad-day',\n", " 'pooh-bear',\n", " 'body-image',\n", " 'immature',\n", " 'secrets',\n", " 'life-and-living',\n", " 'crime',\n", " 'self-denial',\n", " 'frodo',\n", " 'westerfeld',\n", " 'sharing',\n", " 'novelty',\n", " 'crying',\n", " 'proverb',\n", " 'barriers',\n", " 'heathcliff',\n", " 'truth',\n", " 'benediction',\n", " 'ardor',\n", " 'rich',\n", " 'sci-fi',\n", " 'city-of-fallen-angels',\n", " 'unseen',\n", " 'crescendo',\n", " 'comedy',\n", " 'dancing',\n", " 'broad-mindedness',\n", " 'cracks',\n", " 'ignorance',\n", " 'boggs',\n", " 'consolation',\n", " 'profound',\n", " 'sweet',\n", " 'war',\n", " 'fountainhead',\n", " 'the-beatrice-letters',\n", " 'valentine-morgenstern',\n", " 'gamgee',\n", " 'lack-of-love',\n", " 'cosmos',\n", " 'values',\n", " 'human-condition',\n", " 'thirst',\n", " 'love',\n", " 'pig',\n", " 'exercise',\n", " 'criticize',\n", " 'memoir',\n", " 'stop',\n", " 'eyes',\n", " 'life-memories',\n", " 'agama',\n", " 'anger',\n", " 'doomed',\n", " 'he-s-not-just-into-you',\n", " 'bibliophilia',\n", " 'resolve',\n", " 'live',\n", " 'benjamin-button',\n", " 'bilbo',\n", " 'intoxication',\n", " 'reputation',\n", " 'speed-of-light',\n", " 'we-are-what-we-read',\n", " 'inner-beauty',\n", " 'ticking',\n", " 'close-mindedness',\n", " 'action',\n", " 'autonomy',\n", " 'the-mortal-instruments',\n", " 'capability',\n", " 'luke',\n", " 'barbarity',\n", " 'source-unknown',\n", " 'inspirational-ship-storms',\n", " 'power-of-words',\n", " 'knowing',\n", " 'progressive',\n", " 'logo',\n", " 'winnie-the-pooh',\n", " 'friend',\n", " 'extremely',\n", " 'dharma',\n", " 'intellect',\n", " 'psychological',\n", " 'apocalypse',\n", " 'be-yourself',\n", " 'man',\n", " 'dislike',\n", " 'possibilities',\n", " 'purpose-of-life',\n", " 'wishes',\n", " 'strategy',\n", " 'adults',\n", " 'reality-of-life',\n", " 'cleverness',\n", " 'read',\n", " 'mr-darcy',\n", " 'bookstore',\n", " 'fashion',\n", " 'holden-caulfield',\n", " 'before-i-fall',\n", " 'coldness',\n", " 'family',\n", " 'call',\n", " 'abe',\n", " 'self-improvement',\n", " 'on-writing',\n", " 'witches',\n", " 'living-now',\n", " 'invention',\n", " 'gender',\n", " 'reap',\n", " 'misattributed-tom-bodett',\n", " 'snow',\n", " 'self-empowerment',\n", " 'ron',\n", " 'gold',\n", " 'positive-affirmation',\n", " 'calling',\n", " 'relationship',\n", " 'thievery',\n", " 'elizabeth-bennet',\n", " 'morals',\n", " 'role-models',\n", " 'broken-glass',\n", " 'self-reliance',\n", " 'angel',\n", " 'never-give-up',\n", " 'paraphrased',\n", " 'anne-rice',\n", " 'talents',\n", " 'strigoi',\n", " 'seperate',\n", " 'meaningful-life',\n", " 'imagination',\n", " 'reality-bothered',\n", " 'april',\n", " 'city-of-glass',\n", " 'defend',\n", " 'warrior',\n", " 'brave-new-world',\n", " 'forgiveness',\n", " 'dirty',\n", " 'litany-against-fear',\n", " 'uplifting',\n", " 'common-experience',\n", " 'humor-satan',\n", " 'aristotle',\n", " 'antoine-de-saint-exupery',\n", " 'letters',\n", " 'falling-apart',\n", " 'misattributed-to-a-a-milne',\n", " 'grief',\n", " 'immortals',\n", " 'self-awareness',\n", " 'self-responsibility',\n", " 'perseverance',\n", " 'inertia',\n", " 'dating-advice',\n", " 'leadership',\n", " 'awesome-quote',\n", " 'holiness',\n", " 'katniss',\n", " 'dictionary',\n", " 'insincerity',\n", " 'lateral-thinking',\n", " 'fifty-shades-of-grey',\n", " 'forcasting',\n", " 'laughter',\n", " 'james-carstairs',\n", " 'discworld',\n", " 'everyday-life',\n", " 'kindness',\n", " 'rue',\n", " 'jesper',\n", " 'bestseller',\n", " 'struggle',\n", " 'shelter',\n", " 'trap',\n", " 'proactivity',\n", " 'self-love',\n", " 'the-book-thief',\n", " 'deadlines',\n", " 'self-sufficiency',\n", " 'risk-taking',\n", " 'sins',\n", " 'misery',\n", " 'song',\n", " 'chapter-24',\n", " 'you-are-what-you-eat',\n", " 'injury',\n", " 'weight',\n", " 'shampoo',\n", " 'focus',\n", " 'hopelessness',\n", " 'british',\n", " 'breath',\n", " 'women-mothers',\n", " 'misattributed-apple',\n", " 'mentoring',\n", " 'self-help',\n", " 'dursley',\n", " 'harry',\n", " 'democracy',\n", " 'poetry',\n", " 'lover',\n", " 'inspirational-quote',\n", " 'fairytales',\n", " 'loneliness',\n", " 'gossip',\n", " 'parents',\n", " 'art',\n", " 'refuge',\n", " 'bad',\n", " 'teens',\n", " 'hush',\n", " 'privacy',\n", " 'call-me-by-your-name',\n", " 'augustus-waters',\n", " 'reflect',\n", " 'paraphrase-of-lewis-carroll',\n", " 'roses',\n", " 'stupid',\n", " 'deep-thoughts',\n", " 'misattributed-cassandra-clare',\n", " 'moon',\n", " 'trust',\n", " 'mirror',\n", " 'teach',\n", " 'apathy',\n", " 'box',\n", " 'geography',\n", " 'herondale',\n", " 'all-the-bright-places',\n", " 'attention',\n", " 'shadowhunters',\n", " 'scholars',\n", " 'realism',\n", " 'nature',\n", " 'majority',\n", " 'importance',\n", " 'rewind',\n", " 'fudge',\n", " 'deeply-loved',\n", " 'young',\n", " 'lincoln',\n", " 'castle',\n", " 'truth-lies',\n", " 'ya',\n", " 'hg2g',\n", " 'being-loved',\n", " 'grammar',\n", " 'people',\n", " 'opinions',\n", " 'nationalism',\n", " 'talent',\n", " 'language',\n", " 'katniss-everdeen',\n", " 'lamp',\n", " 'forgive',\n", " 'mortal-instruments',\n", " 'nonconformity',\n", " 'lack-of-friendship',\n", " 'do-wrong',\n", " 'songs',\n", " 'stealth',\n", " 'women',\n", " 'girl',\n", " 'mad',\n", " 'noel',\n", " 'possibility',\n", " 'assumptions',\n", " 'money',\n", " 'posiedon',\n", " 'fears',\n", " 'bible',\n", " 'appearance',\n", " 'yearning',\n", " 'self-loathing',\n", " 'choice',\n", " 'confused',\n", " 'detection',\n", " 'going-on',\n", " 'positivity',\n", " 'alice-in-wonderland',\n", " 'reinvention',\n", " 'imperfect',\n", " 'reading-secrets',\n", " 'elite',\n", " 'wings',\n", " 'true-family',\n", " 'mockingjay',\n", " 'write',\n", " 'learn',\n", " 'positive',\n", " 'cats',\n", " 'tedious',\n", " 'katharine-hepburn',\n", " 'ideas',\n", " 'those-who-love-us',\n", " 'confidences',\n", " 'children',\n", " 'censorship',\n", " 'paulo-coelho',\n", " 'changes',\n", " 'earnestness',\n", " 'annabeth-chase',\n", " 'single',\n", " 'name',\n", " 'nicholas',\n", " 'london',\n", " 'tolstoy',\n", " 'good-evil',\n", " 'misattributed-george-santayana',\n", " 'treasure',\n", " 'cynicism',\n", " 'birds',\n", " 'acts-of-kindness',\n", " 'shame',\n", " 'being-yourself',\n", " 'satisfaction',\n", " 'own',\n", " 'belief',\n", " 'throne-of-glass',\n", " 'let-go',\n", " 'attributed',\n", " 'rosemarie-hathaway',\n", " 'eastern-philosophy',\n", " 'oppression',\n", " 'misattributed-audrey-hepburn',\n", " 'longings',\n", " 'thumb',\n", " 'mortality',\n", " 'couples',\n", " 'bella-swan',\n", " 'stay-focused',\n", " 'misattributed-william-g-t-shedd',\n", " 'rationality',\n", " 'toilet',\n", " 'thalia',\n", " 'martyrdom',\n", " 'clean',\n", " 'mistakes',\n", " 'self-belief',\n", " 'sleep',\n", " 'zen',\n", " 'inner-life',\n", " 'bene-gesserit',\n", " 'anne',\n", " 'america',\n", " 'self-irony',\n", " 'rebirth',\n", " 'explore',\n", " 'sense-of-wonder',\n", " 'sing',\n", " 'elitism',\n", " 'abilities',\n", " 'wilde',\n", " 'edison',\n", " 'amor-deliria-nervosa',\n", " 'writing-from-the-heart',\n", " 'the-selection',\n", " 'self-respect',\n", " 'color',\n", " 'dramatist-quotes',\n", " 'ingenuity',\n", " 'meir',\n", " 'drug',\n", " 'dissapointment',\n", " 'wither',\n", " 'broccoli',\n", " 'the-way',\n", " 'mortification',\n", " 'poets',\n", " 'planning',\n", " 'separation',\n", " 'inej',\n", " 'writer',\n", " 'sympathy',\n", " 'crazy',\n", " 'individualism',\n", " 'equations',\n", " 'apothegm',\n", " 'misattributed-gloria-steinem',\n", " 'right-way',\n", " 'pijamas',\n", " 'fortitude',\n", " 'protection',\n", " 'arts-and-humanities',\n", " '1929',\n", " 'unhappiness',\n", " 'june',\n", " 'to-be-loved',\n", " 'quote-investigator',\n", " 'liar',\n", " 'panic',\n", " 'authenticity',\n", " 'shakespeare',\n", " 'inspirational-stories',\n", " 'thought',\n", " 'greek',\n", " 'mythological',\n", " 'lock-and-key',\n", " 'misattributed-kurt-cobain',\n", " 'failures',\n", " 'fame',\n", " 'stephen-chbosky',\n", " 'breakup',\n", " 'of',\n", " 'inspirational-love',\n", " 'true-to-self',\n", " 'company',\n", " 'hunger-games-peeta-katniss',\n", " 'authentic-self',\n", " 'contrast',\n", " 'humility',\n", " 'islam',\n", " 'organized-religion',\n", " 'christmas',\n", " 'exploration',\n", " 'unhappy-life',\n", " 'sun',\n", " 'last-sacrifice',\n", " 'friends',\n", " 'civil-rights',\n", " 'adversity',\n", " 'trouble',\n", " 'music',\n", " 'day',\n", " 'afraid',\n", " 'meyer',\n", " 'diary-entry',\n", " 'character',\n", " 'self-worth',\n", " 'ian',\n", " 'sal-paradise',\n", " 'misattributed-john-wayne',\n", " 'happy-endings',\n", " 'spiritual-growth',\n", " 'henry-miller',\n", " 'wizard',\n", " 'being-human',\n", " 'jared',\n", " 'morgenstern',\n", " 'olympians',\n", " 'challanges',\n", " 'gender-stereotypes',\n", " 'dracula',\n", " 'albus-potter',\n", " 'self-control',\n", " 'flexibility',\n", " 'akeelah',\n", " 'yesterday',\n", " 'jk-rowling',\n", " 'understanding-others',\n", " 'misattributed-to-abraham-lincoln',\n", " 'pride',\n", " 'chivalry',\n", " 'courage',\n", " 'move-on',\n", " 'adulthood',\n", " 'love-yourself',\n", " 'challenge',\n", " 'james-potter',\n", " 'layken',\n", " 'propaganda',\n", " 'kiss',\n", " 'amelia-peabody',\n", " 'joh-green',\n", " 'canoodle',\n", " 'samwise',\n", " 'sadness',\n", " 'alec',\n", " 'impossible',\n", " 'misattributed-jace',\n", " 'childhood-memory',\n", " 'melanie',\n", " 'blessed',\n", " 'greedy',\n", " 'twilight-saga',\n", " 'plays',\n", " 'fool',\n", " 'worst',\n", " 'anais-nin',\n", " 'knowledge-power',\n", " 'open-minds',\n", " 'rhysand',\n", " 'rebellion',\n", " 'regret',\n", " 'grudges',\n", " 'novelist',\n", " 'forever',\n", " 'most-famous-novella',\n", " 'timidity',\n", " 'reincarnation',\n", " 'eccentricity',\n", " 'gullibility',\n", " 'believe',\n", " 'isolation',\n", " 'park',\n", " 'integrity-quotes',\n", " 'laugh',\n", " 'atoms',\n", " ...}" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "alltags = set()\n", "\n", "for l in df['tags']:\n", " for item in l:\n", " alltags.add(item) \n", "\n", "alltags" ] }, { "cell_type": "code", "execution_count": 18, "id": "c8030228", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "2161" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(alltags)" ] }, { "cell_type": "markdown", "id": "9d7208a0", "metadata": {}, "source": [ "This is quite a lot of tags, many of which are fairly obscure. Many of the obscure tags consist of multiple words joined by dashes. We can isolate all of the single word tags to get a shorter set." ] }, { "cell_type": "code", "execution_count": 19, "id": "1f83a950", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'romantic',\n", " 'speak',\n", " 'mystery',\n", " 'feelings',\n", " 'hazel',\n", " 'hobbit',\n", " 'friendship',\n", " 'vorkosigan',\n", " 'odair',\n", " 'taste',\n", " 'gaiman',\n", " 'improvement',\n", " 'nerdfighter',\n", " 'dystopia',\n", " 'renewal',\n", " 'arts',\n", " 'jace',\n", " 'david',\n", " 'minister',\n", " 'christina',\n", " 'hate',\n", " 'inspiration',\n", " 'discipline',\n", " 'brain',\n", " 'insults',\n", " 'wormtail',\n", " 'facts',\n", " 'worry',\n", " 'sexuality',\n", " 'musical',\n", " 'summer',\n", " 'pretend',\n", " 'habits',\n", " 'automobile',\n", " 'why',\n", " 'soul',\n", " 'desire',\n", " 'light',\n", " 'innocence',\n", " '2012',\n", " 'danger',\n", " 'richness',\n", " 'hidden',\n", " 'unity',\n", " 'dreaming',\n", " 'principles',\n", " 'hippy',\n", " 'mottos',\n", " 'purpose',\n", " 'tea',\n", " 'dream',\n", " 'touching',\n", " 'wealth',\n", " 'enemies',\n", " 'aloneness',\n", " 'leaving',\n", " 'inspire',\n", " 'tobias',\n", " 'relaxation',\n", " 'unfairness',\n", " 'relationships',\n", " 'romance',\n", " 'names',\n", " 'pretty',\n", " 'forgetting',\n", " 'accomplishment',\n", " 'diversity',\n", " 'nightmares',\n", " 'right',\n", " 'security',\n", " 'golda',\n", " 'happy',\n", " 'clock',\n", " 'coffee',\n", " 'femininity',\n", " 'literacy',\n", " 'heartbreak',\n", " 'serenity',\n", " 'ataraxy',\n", " 'awesome',\n", " 'buddhism',\n", " 'patriotism',\n", " 'share',\n", " 'television',\n", " 'opportunity',\n", " 'person',\n", " 'agree',\n", " 'lies',\n", " 'towns',\n", " 'spiritual',\n", " 'thinking',\n", " 'save',\n", " 'defenses',\n", " 'flattery',\n", " 'thing',\n", " 'tyson',\n", " 'science',\n", " 'adventuring',\n", " 'cows',\n", " 'quotation',\n", " 'frolick',\n", " 'listen',\n", " 'expression',\n", " 'four',\n", " 'behind',\n", " 'soon',\n", " '1955',\n", " 'ruby',\n", " 'hurt',\n", " 'force',\n", " 'everything',\n", " 'humor',\n", " 'silence',\n", " 'death',\n", " 'teasing',\n", " 'mother',\n", " 'attraction',\n", " 'guys',\n", " 'advocacy',\n", " 'skeptic',\n", " 'winter',\n", " 'feminism',\n", " 'beatrice',\n", " 'introspection',\n", " 'companionship',\n", " 'dionysus',\n", " 'doing',\n", " 'finnick',\n", " 'melancholy',\n", " 'parting',\n", " 'fuck',\n", " 'snape',\n", " 'owen',\n", " 'frank',\n", " 'bossypants',\n", " 'creation',\n", " 'rationalism',\n", " 'misattributed',\n", " 'concern',\n", " 'inkheart',\n", " 'honor',\n", " 'sausage',\n", " 'clouds',\n", " 'mindful',\n", " 'evidence',\n", " 'metaphor',\n", " 'sufi',\n", " 'success',\n", " 'memory',\n", " 'thoughts',\n", " 'haunt',\n", " 'acorns',\n", " 'valour',\n", " 'sweetness',\n", " 'will',\n", " 'giving',\n", " 'responsibility',\n", " 'caring',\n", " 'goodbye',\n", " 'loner',\n", " 'sad',\n", " 'absence',\n", " 'judge',\n", " 'act',\n", " 'jail',\n", " 'consul',\n", " 'ferret',\n", " 'sea',\n", " 'secret',\n", " 'sin',\n", " 'missing',\n", " 'unsourced',\n", " 'madness',\n", " 'troublemakers',\n", " 'neverland',\n", " 'lookingforalaska',\n", " 'blood',\n", " 'sebastian',\n", " 'enthusiasm',\n", " 'risk',\n", " 'isabelle',\n", " 'nagasawa',\n", " 'physics',\n", " 'parties',\n", " 'wisher',\n", " 'tired',\n", " 'psychotherapist',\n", " 'agnosticism',\n", " 'zeus',\n", " 'miracle',\n", " 'seeing',\n", " 'yourself',\n", " 'passion',\n", " 'shock',\n", " 'cinna',\n", " 'assurance',\n", " 'safety',\n", " 'society',\n", " 'fire',\n", " 'sow',\n", " 'misprints',\n", " 'eternity',\n", " 'rescuing',\n", " 'rythmic',\n", " 'sense',\n", " 'ideals',\n", " 'neighborhoods',\n", " 'wrong',\n", " 'congratulations',\n", " 'singer',\n", " 'waiting',\n", " 'teenagers',\n", " 'philosphy',\n", " 'strength',\n", " 'principle',\n", " 'eragon',\n", " 'give',\n", " 'feathers',\n", " 'darkness',\n", " 'age',\n", " 'haunting',\n", " 'christians',\n", " 'simple',\n", " 'chinese',\n", " 'painting',\n", " 'loud',\n", " 'homicide',\n", " 'judgment',\n", " 'adoration',\n", " 'troubles',\n", " 'quotes',\n", " 'dreams',\n", " 'dark',\n", " 'talking',\n", " 'morning',\n", " 'excuse',\n", " 'nonfiction',\n", " 'inspirational',\n", " 'simile',\n", " 'nin',\n", " 'ghost',\n", " 'gandalf',\n", " 'home',\n", " 'wife',\n", " 'vanity',\n", " 'encouragement',\n", " 'sirius',\n", " 'learning',\n", " 'vlog',\n", " 'skepticism',\n", " 'forgiving',\n", " 'americans',\n", " 'atheist',\n", " 'empowerment',\n", " 'smiles',\n", " 'whiskey',\n", " 'humans',\n", " 'blues',\n", " 'openness',\n", " 'experience',\n", " 'books',\n", " 'shrugged',\n", " 'marriage',\n", " 'incrementalism',\n", " 'stories',\n", " 'fiction',\n", " 'obvious',\n", " 'individuality',\n", " 'december',\n", " 'ordinary',\n", " 'chocolate',\n", " 'disappointment',\n", " 'tfios',\n", " 'santiago',\n", " 'judgement',\n", " 'busy',\n", " 'failure',\n", " 'betrayal',\n", " 'awe',\n", " 'atheism',\n", " 'distance',\n", " 'mermaids',\n", " 'heartwarming',\n", " 'hypocrisy',\n", " 'loyalty',\n", " 'witch',\n", " 'rebels',\n", " 'gandhi',\n", " 'commercial',\n", " 'telekinetics',\n", " 'dragons',\n", " 'kissing',\n", " 'deceit',\n", " 'smile',\n", " 'procrastinate',\n", " 'confession',\n", " 'jokes',\n", " 'men',\n", " 'famous',\n", " 'social',\n", " 'anais',\n", " 'lemons',\n", " 'hope',\n", " 'mormont',\n", " 'originality',\n", " 'glass',\n", " 'devil',\n", " 'dizziness',\n", " 'education',\n", " 'rulers',\n", " 'attitude',\n", " 'feminist',\n", " 'deception',\n", " 'literature',\n", " 'girls',\n", " 'fun',\n", " 'mind',\n", " 'lupin',\n", " 'thorns',\n", " 'pets',\n", " 'lust',\n", " 'misanthropy',\n", " 'energy',\n", " 'challenges',\n", " 'pleasure',\n", " 'gift',\n", " 'paradox',\n", " 'unpredictability',\n", " 'wring',\n", " 'independence',\n", " 'dreamer',\n", " 'equality',\n", " 'extraterrestrials',\n", " 'safran',\n", " 'vegetarianism',\n", " 'nerds',\n", " 'requirements',\n", " 'silliness',\n", " 'immature',\n", " 'secrets',\n", " 'crime',\n", " 'frodo',\n", " 'westerfeld',\n", " 'sharing',\n", " 'novelty',\n", " 'crying',\n", " 'proverb',\n", " 'barriers',\n", " 'heathcliff',\n", " 'truth',\n", " 'benediction',\n", " 'ardor',\n", " 'rich',\n", " 'unseen',\n", " 'crescendo',\n", " 'comedy',\n", " 'dancing',\n", " 'cracks',\n", " 'ignorance',\n", " 'boggs',\n", " 'consolation',\n", " 'profound',\n", " 'sweet',\n", " 'war',\n", " 'fountainhead',\n", " 'gamgee',\n", " 'cosmos',\n", " 'values',\n", " 'thirst',\n", " 'love',\n", " 'pig',\n", " 'exercise',\n", " 'criticize',\n", " 'memoir',\n", " 'stop',\n", " 'eyes',\n", " 'agama',\n", " 'anger',\n", " 'doomed',\n", " 'bibliophilia',\n", " 'resolve',\n", " 'live',\n", " 'bilbo',\n", " 'intoxication',\n", " 'reputation',\n", " 'ticking',\n", " 'action',\n", " 'autonomy',\n", " 'capability',\n", " 'luke',\n", " 'barbarity',\n", " 'knowing',\n", " 'progressive',\n", " 'logo',\n", " 'friend',\n", " 'extremely',\n", " 'dharma',\n", " 'intellect',\n", " 'psychological',\n", " 'apocalypse',\n", " 'man',\n", " 'dislike',\n", " 'possibilities',\n", " 'wishes',\n", " 'strategy',\n", " 'adults',\n", " 'cleverness',\n", " 'read',\n", " 'bookstore',\n", " 'fashion',\n", " 'coldness',\n", " 'family',\n", " 'call',\n", " 'abe',\n", " 'witches',\n", " 'invention',\n", " 'gender',\n", " 'reap',\n", " 'snow',\n", " 'ron',\n", " 'gold',\n", " 'calling',\n", " 'relationship',\n", " 'thievery',\n", " 'morals',\n", " 'angel',\n", " 'paraphrased',\n", " 'talents',\n", " 'strigoi',\n", " 'seperate',\n", " 'imagination',\n", " 'april',\n", " 'defend',\n", " 'warrior',\n", " 'forgiveness',\n", " 'dirty',\n", " 'uplifting',\n", " 'aristotle',\n", " 'letters',\n", " 'grief',\n", " 'immortals',\n", " 'leadership',\n", " 'perseverance',\n", " 'inertia',\n", " 'holiness',\n", " 'katniss',\n", " 'dictionary',\n", " 'insincerity',\n", " 'forcasting',\n", " 'laughter',\n", " 'discworld',\n", " 'kindness',\n", " 'rue',\n", " 'jesper',\n", " 'bestseller',\n", " 'struggle',\n", " 'shelter',\n", " 'trap',\n", " 'proactivity',\n", " 'deadlines',\n", " 'sins',\n", " 'misery',\n", " 'song',\n", " 'injury',\n", " 'weight',\n", " 'shampoo',\n", " 'focus',\n", " 'hopelessness',\n", " 'british',\n", " 'breath',\n", " 'mentoring',\n", " 'dursley',\n", " 'harry',\n", " 'democracy',\n", " 'poetry',\n", " 'lover',\n", " 'fairytales',\n", " 'loneliness',\n", " 'gossip',\n", " 'parents',\n", " 'art',\n", " 'refuge',\n", " 'bad',\n", " 'teens',\n", " 'hush',\n", " 'privacy',\n", " 'reflect',\n", " 'roses',\n", " 'stupid',\n", " 'moon',\n", " 'trust',\n", " 'mirror',\n", " 'teach',\n", " 'apathy',\n", " 'box',\n", " 'geography',\n", " 'herondale',\n", " 'attention',\n", " 'shadowhunters',\n", " 'scholars',\n", " 'realism',\n", " 'nature',\n", " 'majority',\n", " 'importance',\n", " 'rewind',\n", " 'fudge',\n", " 'young',\n", " 'lincoln',\n", " 'castle',\n", " 'ya',\n", " 'hg2g',\n", " 'grammar',\n", " 'people',\n", " 'opinions',\n", " 'nationalism',\n", " 'talent',\n", " 'language',\n", " 'lamp',\n", " 'forgive',\n", " 'nonconformity',\n", " 'songs',\n", " 'stealth',\n", " 'women',\n", " 'girl',\n", " 'mad',\n", " 'noel',\n", " 'possibility',\n", " 'assumptions',\n", " 'money',\n", " 'posiedon',\n", " 'fears',\n", " 'bible',\n", " 'appearance',\n", " 'yearning',\n", " 'choice',\n", " 'confused',\n", " 'detection',\n", " 'positivity',\n", " 'reinvention',\n", " 'imperfect',\n", " 'elite',\n", " 'wings',\n", " 'mockingjay',\n", " 'write',\n", " 'learn',\n", " 'positive',\n", " 'cats',\n", " 'tedious',\n", " 'ideas',\n", " 'confidences',\n", " 'children',\n", " 'censorship',\n", " 'changes',\n", " 'earnestness',\n", " 'single',\n", " 'name',\n", " 'nicholas',\n", " 'london',\n", " 'tolstoy',\n", " 'treasure',\n", " 'cynicism',\n", " 'birds',\n", " 'shame',\n", " 'satisfaction',\n", " 'own',\n", " 'belief',\n", " 'attributed',\n", " 'oppression',\n", " 'longings',\n", " 'thumb',\n", " 'mortality',\n", " 'couples',\n", " 'rationality',\n", " 'toilet',\n", " 'thalia',\n", " 'martyrdom',\n", " 'clean',\n", " 'mistakes',\n", " 'sleep',\n", " 'zen',\n", " 'anne',\n", " 'america',\n", " 'rebirth',\n", " 'explore',\n", " 'sing',\n", " 'elitism',\n", " 'abilities',\n", " 'wilde',\n", " 'edison',\n", " 'color',\n", " 'ingenuity',\n", " 'meir',\n", " 'drug',\n", " 'dissapointment',\n", " 'wither',\n", " 'broccoli',\n", " 'mortification',\n", " 'poets',\n", " 'planning',\n", " 'separation',\n", " 'inej',\n", " 'writer',\n", " 'sympathy',\n", " 'crazy',\n", " 'individualism',\n", " 'equations',\n", " 'apothegm',\n", " 'pijamas',\n", " 'fortitude',\n", " 'protection',\n", " '1929',\n", " 'unhappiness',\n", " 'june',\n", " 'liar',\n", " 'panic',\n", " 'authenticity',\n", " 'shakespeare',\n", " 'thought',\n", " 'greek',\n", " 'mythological',\n", " 'failures',\n", " 'fame',\n", " 'breakup',\n", " 'of',\n", " 'company',\n", " 'contrast',\n", " 'humility',\n", " 'islam',\n", " 'christmas',\n", " 'exploration',\n", " 'sun',\n", " 'friends',\n", " 'adversity',\n", " 'trouble',\n", " 'music',\n", " 'day',\n", " 'afraid',\n", " 'meyer',\n", " 'character',\n", " 'ian',\n", " 'wizard',\n", " 'jared',\n", " 'morgenstern',\n", " 'olympians',\n", " 'challanges',\n", " 'dracula',\n", " 'flexibility',\n", " 'akeelah',\n", " 'yesterday',\n", " 'pride',\n", " 'chivalry',\n", " 'courage',\n", " 'adulthood',\n", " 'challenge',\n", " 'layken',\n", " 'propaganda',\n", " 'kiss',\n", " 'canoodle',\n", " 'samwise',\n", " 'sadness',\n", " 'alec',\n", " 'impossible',\n", " 'melanie',\n", " 'blessed',\n", " 'greedy',\n", " 'plays',\n", " 'fool',\n", " 'worst',\n", " 'rhysand',\n", " 'rebellion',\n", " 'regret',\n", " 'grudges',\n", " 'novelist',\n", " 'forever',\n", " 'timidity',\n", " 'reincarnation',\n", " 'eccentricity',\n", " 'gullibility',\n", " 'believe',\n", " 'isolation',\n", " 'park',\n", " 'laugh',\n", " 'atoms',\n", " 'beatles',\n", " 'satirical',\n", " 'freethinker',\n", " 'neurosis',\n", " 'sinister',\n", " 'determination',\n", " 'identity',\n", " 'emptiness',\n", " 'antipathy',\n", " 'house',\n", " 'law',\n", " 'memories',\n", " 'paradise',\n", " 'carroll',\n", " 'einstein',\n", " 'psychology',\n", " 'college',\n", " 'warriors',\n", " 'beginnings',\n", " 'husband',\n", " 'visions',\n", " 'earth',\n", " 'seuss',\n", " 'salinger',\n", " 'others',\n", " 'trauma',\n", " 'conversations',\n", " 'bella',\n", " 'parks',\n", " 'beatnik',\n", " 'uncertainty',\n", " 'politics',\n", " 'unburdening',\n", " 'question',\n", " 'miracles',\n", " 'unbreakable',\n", " 'difficult',\n", " 'watch',\n", " 'intimidation',\n", " 'humorous',\n", " 'divergent',\n", " 'destruction',\n", " 'cleanliness',\n", " 'heroes',\n", " 'voice',\n", " 'recollection',\n", " 'allegory',\n", " 'service',\n", " 'pooh',\n", " 'stressed',\n", " 'wayland',\n", " 'evil',\n", " 'poem',\n", " 'disease',\n", " 'loss',\n", " 'films',\n", " 'eccentric',\n", " 'mutuality',\n", " 'heartbreaking',\n", " 'machines',\n", " 'night',\n", " 'charming',\n", " 'catatonia',\n", " 'opposite',\n", " 'amnesia',\n", " 'thankful',\n", " 'optimistic',\n", " 'creative',\n", " 'seduction',\n", " 'goodbyes',\n", " 'permission',\n", " '2008',\n", " 'poseidon',\n", " 'grow',\n", " 'mouth',\n", " 'teacher',\n", " 'sinners',\n", " 'complaining',\n", " 'richelle',\n", " 'opposition',\n", " 'pedophilia',\n", " 'amsterdam',\n", " 'flame',\n", " 'healing',\n", " 'respect',\n", " 'touchstones',\n", " 'bereavement',\n", " 'charity',\n", " 'mindfulness',\n", " 'jealousy',\n", " 'clothing',\n", " 'beauty',\n", " 'true',\n", " 'human',\n", " 'purity',\n", " 'blackjack',\n", " 'faith',\n", " 'pigs',\n", " 'dog',\n", " 'feeling',\n", " 'dissent',\n", " 'virtues',\n", " 'funeral',\n", " 'plant',\n", " 'radiance',\n", " 'mail',\n", " 'depressing',\n", " 'inspiring',\n", " 'drugs',\n", " 'sensation',\n", " 'intelligence',\n", " 'misfits',\n", " 'arrow',\n", " 'shine',\n", " 'humour',\n", " 'internet',\n", " 'nothing',\n", " 'perception',\n", " 'governments',\n", " 'acomaf',\n", " 'prank',\n", " 'ambiguity',\n", " 'selfishness',\n", " 'obsession',\n", " 'puck',\n", " 'decisions',\n", " 'blog',\n", " 'boldness',\n", " 'intent',\n", " 'illusions',\n", " 'cynical',\n", " 'navigation',\n", " 'poet',\n", " 'protest',\n", " 'mythology',\n", " 'curiosity',\n", " 'dating',\n", " 'sunset',\n", " 'stupidity',\n", " 'goodness',\n", " 'seasons',\n", " 'good',\n", " 'wind',\n", " 'wayfinding',\n", " 'personality',\n", " 'joke',\n", " 'library',\n", " '1993',\n", " 'achievement',\n", " 'scotland',\n", " 'h2g2',\n", " 'religion',\n", " 'doubt',\n", " 'idleness',\n", " 'empathy',\n", " 'woods',\n", " 'sandman',\n", " 'killing',\n", " 'obviousness',\n", " 'celebrity',\n", " 'fearless',\n", " 'self',\n", " 'twilight',\n", " 'characters',\n", " 'imposters',\n", " 'irony',\n", " 'humanity',\n", " 'amity',\n", " 'perspective',\n", " 'expectations',\n", " 'addiction',\n", " 'witchcraft',\n", " 'birthday',\n", " 'frost',\n", " 'trees',\n", " 'jim',\n", " 'sarcasm',\n", " 'dna',\n", " 'conformity',\n", " 'choices',\n", " 'crisis',\n", " 'diary',\n", " 'freedom',\n", " 'apple',\n", " 'youth',\n", " 'ambition',\n", " 'morality',\n", " 'teaching',\n", " 'peace',\n", " 'idiocy',\n", " 'kill',\n", " 'integrity',\n", " 'pacifism',\n", " 'description',\n", " 'optimism',\n", " 'promise',\n", " 'misquote',\n", " 'grandparents',\n", " 'clichés',\n", " 'ugly',\n", " 'aesthetics',\n", " 'superiority',\n", " 'quip',\n", " 'belonging',\n", " 'change',\n", " 'lecture',\n", " 'inevitability',\n", " 'storytelling',\n", " 'divorce',\n", " 'remembrance',\n", " 'misogyny',\n", " 'moi',\n", " 'inconvenient',\n", " 'grudge',\n", " 'appreciate',\n", " 'plants',\n", " 'clary',\n", " 'puppies',\n", " 'stele',\n", " 'universe',\n", " 'scars',\n", " 'wonder',\n", " 'tomorrow',\n", " 'consciousness',\n", " 'profanity',\n", " 'survival',\n", " 'winnie',\n", " 'advertising',\n", " 'delusion',\n", " 'essence',\n", " 'fred',\n", " 'spirituality',\n", " 'safe',\n", " 'disappointed',\n", " 'school',\n", " 'mask',\n", " 'reading',\n", " 'letter',\n", " 'fight',\n", " 'amorality',\n", " 'popularity',\n", " 'incredibly',\n", " 'together',\n", " 'desperate',\n", " 'late',\n", " 'makeup',\n", " 'lying',\n", " 'different',\n", " 'answers',\n", " 'beatitudes',\n", " 'limitations',\n", " 'flewn',\n", " 'wounds',\n", " 'adaptation',\n", " 'jesus',\n", " 'dying',\n", " 'magnificent',\n", " 'awkward',\n", " 'myself',\n", " 'fair',\n", " 'connecting',\n", " 'intellectuals',\n", " 'worship',\n", " 'evermore',\n", " 'procrastination',\n", " 'temptation',\n", " 'werewolf',\n", " 'longing',\n", " 'haiku',\n", " 'expectation',\n", " 'possiblity',\n", " 'hunger',\n", " 'forgetfulness',\n", " 'foolproof',\n", " 'happiness',\n", " 'arrogance',\n", " 'punishment',\n", " 'imperfections',\n", " 'computers',\n", " 'stake',\n", " 'losing',\n", " 'criticism',\n", " 'anxiety',\n", " 'adventure',\n", " 'perfection',\n", " 'ocean',\n", " 'demons',\n", " 'cartoonist',\n", " 'chemistry',\n", " 'heart',\n", " 'awesomeness',\n", " 'spinning',\n", " 'macbeth',\n", " 'listening',\n", " 'ache',\n", " 'museums',\n", " 'dogs',\n", " 'motivational',\n", " 'acceptance',\n", " 'elections',\n", " 'trying',\n", " 'reggae',\n", " 'sorry',\n", " 'conscience',\n", " 'scared',\n", " 'leader',\n", " 'nico',\n", " 'dangerous',\n", " 'jamie',\n", " 'vision',\n", " 'ethos',\n", " 'rocks',\n", " 'nerd',\n", " 'commerce',\n", " 'reason',\n", " 'astronomer',\n", " 'academy',\n", " ...}" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "simpletags = set()\n", "\n", "for tag in alltags:\n", " if tag.find('-') == -1:\n", " simpletags.add(tag)\n", "\n", "simpletags" ] }, { "cell_type": "code", "execution_count": 20, "id": "2b0c0ef1", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1518" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(simpletags)" ] }, { "cell_type": "markdown", "id": "37fad9e6", "metadata": {}, "source": [ "Another thing we can do is to isolate quotes by a single tag. The code below shows how to isolate just those quotes marked with the tag 'books'." ] }, { "cell_type": "code", "execution_count": 3, "id": "2e05447e", "metadata": {}, "outputs": [], "source": [ "def tagContains(row,term):\n", " if term in row['tags']:\n", " return True\n", " else:\n", " return False\n", "\n", "def filterFrame(data,category):\n", " return df[df.apply(lambda row: tagContains(row,category), axis=1)]" ] }, { "cell_type": "code", "execution_count": 4, "id": "a4b21edf", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
quoteauthortags
3“So many books, so little time.”Frank Zappa[books, humor]
4“A room without books is like a body without a...Marcus Tullius Cicero[books, simile, soul]
32“The person, be it gentleman or lady, who has ...Jane Austen,[aliteracy, books, classic, humor]
36“Good friends, good books, and a sleepy consci...Mark Twain[books, contentment, friends, friendship, life]
39“Fairy tales are more than true: not because t...Neil Gaiman,[books, dragons, fairy-tales, inspirational, p...
............
2326“Five exclamation marks, the sure sign of an i...Terry Pratchett,[books, humor]
2330“Reading is like thinking, like praying, like ...Roberto Bolaño,[books, reading]
2345“Literature is a luxury; fiction is a necessity.”G.K. Chesterton[books, literature, on-fiction]
2484“The worst thing about new books is that they ...Joseph Joubert[books, reading]
2488“The VoiceThere is a voice inside of youThat w...Shel Silverstein[books, inner-voice, leaders, parents, right, ...
\n", "

141 rows × 3 columns

\n", "
" ], "text/plain": [ " quote \\\n", "3 “So many books, so little time.” \n", "4 “A room without books is like a body without a... \n", "32 “The person, be it gentleman or lady, who has ... \n", "36 “Good friends, good books, and a sleepy consci... \n", "39 “Fairy tales are more than true: not because t... \n", "... ... \n", "2326 “Five exclamation marks, the sure sign of an i... \n", "2330 “Reading is like thinking, like praying, like ... \n", "2345 “Literature is a luxury; fiction is a necessity.” \n", "2484 “The worst thing about new books is that they ... \n", "2488 “The VoiceThere is a voice inside of youThat w... \n", "\n", " author tags \n", "3 Frank Zappa [books, humor] \n", "4 Marcus Tullius Cicero [books, simile, soul] \n", "32 Jane Austen, [aliteracy, books, classic, humor] \n", "36 Mark Twain [books, contentment, friends, friendship, life] \n", "39 Neil Gaiman, [books, dragons, fairy-tales, inspirational, p... \n", "... ... ... \n", "2326 Terry Pratchett, [books, humor] \n", "2330 Roberto Bolaño, [books, reading] \n", "2345 G.K. Chesterton [books, literature, on-fiction] \n", "2484 Joseph Joubert [books, reading] \n", "2488 Shel Silverstein [books, inner-voice, leaders, parents, right, ... \n", "\n", "[141 rows x 3 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "books_df = filterFrame(df,'books')\n", "books_df" ] }, { "cell_type": "markdown", "id": "4786764e", "metadata": {}, "source": [ "This produces a more managable set of quotes, but at 141 quotes this is still a fairly large set.\n", "\n", "Other tags produce similarly large groups of quotes." ] }, { "cell_type": "code", "execution_count": null, "id": "a3dc961b", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
quoteauthortags
2“Two things are infinite: the universe and hum...Albert Einstein[human-nature, humor, infinity, philosophy, sc...
3“So many books, so little time.”Frank Zappa[books, humor]
8“You only live once, but if you do it right, o...Mae West[humor, life]
25“Insanity is doing the same thing, over and ov...Narcotics Anonymous[humor, insanity, life, misattributed-ben-fran...
32“The person, be it gentleman or lady, who has ...Jane Austen,[aliteracy, books, classic, humor]
............
2443“Sebastian just smiled. “I could hear your h...Cassandra Clare,[humor, jace-wayland, realization, sarcasm, se...
2453“Moths,\" repeats Will. \"You're afraid of moths...Veronica Roth,[humor]
2470“If you try to fail, and succeed, which have y...George Carlin[humor, humour, paradox]
2476“If my doctor told me I had only six minutes t...Isaac Asimov[humor, writing]
2482“See, the problem is that God gives men a brai...Robin Williams[humor, men]
\n", "

254 rows × 3 columns

\n", "
" ], "text/plain": [ " quote author \\\n", "2 “Two things are infinite: the universe and hum... Albert Einstein \n", "3 “So many books, so little time.” Frank Zappa \n", "8 “You only live once, but if you do it right, o... Mae West \n", "25 “Insanity is doing the same thing, over and ov... Narcotics Anonymous \n", "32 “The person, be it gentleman or lady, who has ... Jane Austen, \n", "... ... ... \n", "2443 “Sebastian just smiled. “I could hear your h... Cassandra Clare, \n", "2453 “Moths,\" repeats Will. \"You're afraid of moths... Veronica Roth, \n", "2470 “If you try to fail, and succeed, which have y... George Carlin \n", "2476 “If my doctor told me I had only six minutes t... Isaac Asimov \n", "2482 “See, the problem is that God gives men a brai... Robin Williams \n", "\n", " tags \n", "2 [human-nature, humor, infinity, philosophy, sc... \n", "3 [books, humor] \n", "8 [humor, life] \n", "25 [humor, insanity, life, misattributed-ben-fran... \n", "32 [aliteracy, books, classic, humor] \n", "... ... \n", "2443 [humor, jace-wayland, realization, sarcasm, se... \n", "2453 [humor] \n", "2470 [humor, humour, paradox] \n", "2476 [humor, writing] \n", "2482 [humor, men] \n", "\n", "[254 rows x 3 columns]" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "humor_df = filterFrame(df,'humor')\n", "humor_df" ] }, { "cell_type": "markdown", "id": "276cdced", "metadata": {}, "source": [ "### The Assignment\n", "\n", "Your job is to use the dataset as the basis for an epigram generating RAG application. Given a block of text, your job will be to find an appropriate quote from the dataset to use as an epigram for the text.\n", "\n", "Since building RAG applications is more of an art than a science, I am not going to give you a single best approach for this problem. Instead, I am going to give you a set of suggestions that you may or may not choose to use. You will have to experiment with some of these ideas and/or supplement them with ideas of your own.\n", "\n", "Here are some things you can try:\n", "\n", "1. You will want to start by putting all of the quotes in a vector database. For each quote you can store the author and the tags as metadata associated with the quote.\n", "\n", "2. Write some python code to identify the 10 or 20 most popular tags. As part of your application you can ask an LLM to look at the block of text you are working with and then pick a tag from the list of popular tags that the LLM thinks best characterizes the text. You can then use that tag to help filter your search in the vector database. You should research how chromadb allows you to use metadata to filter a database search.\n", "\n", "3. You probably don't want to use the full block of text provided as your query text for the vector database. Instead, you probably want the LLM to summarize the entire block of text or individual paragraphs from the text to give you more focused text you can use as your query text for the vector database.\n", "\n", "4. After getting some candidate quotes from the database, you can ask the LLM to evaluate the quotes and pick the one that works best as the epigram for the full block of text." ] }, { "cell_type": "markdown", "id": "e06d49cb", "metadata": {}, "source": [ "### Important Requirement\n", "\n", "For this assignment you should use the smaller of the two LLMs we have available, the one at 143.44.10.188:1234.\n", "\n", "You are not allowed to solve this problem by simply asking the LLM to produce a quote for you. The quote your system returns should be one of the quotes pulled from the dataset." ] }, { "cell_type": "markdown", "id": "71c8ab89", "metadata": {}, "source": [ "### Helpful Hint\n", "\n", "Here is a page that gives suggestions on how you can apply tags to Chromadb documents as metadata:\n", "\n", "https://cookbook.chromadb.dev/strategies/multi-category-filters/" ] } ], "metadata": { "kernelspec": { "display_name": "agents", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.0" } }, "nbformat": 4, "nbformat_minor": 5 }