| 1 | { |
|---|
| 2 | "nbformat": 4, |
|---|
| 3 | "nbformat_minor": 0, |
|---|
| 4 | "metadata": { |
|---|
| 5 | "colab": { |
|---|
| 6 | "provenance": [], |
|---|
| 7 | "gpuType": "T4", |
|---|
| 8 | "toc_visible": true, |
|---|
| 9 | "collapsed_sections": [ |
|---|
| 10 | "1xAYTR67fp73", |
|---|
| 11 | "ivnqydtMgHpt", |
|---|
| 12 | "x_qd8O0fgs8M" |
|---|
| 13 | ] |
|---|
| 14 | }, |
|---|
| 15 | "kernelspec": { |
|---|
| 16 | "name": "python3", |
|---|
| 17 | "display_name": "Python 3" |
|---|
| 18 | }, |
|---|
| 19 | "language_info": { |
|---|
| 20 | "name": "python" |
|---|
| 21 | } |
|---|
| 22 | }, |
|---|
| 23 | "cells": [ |
|---|
| 24 | { |
|---|
| 25 | "cell_type": "markdown", |
|---|
| 26 | "source": [ |
|---|
| 27 | "# Preparation Steps\n", |
|---|
| 28 | "Install Hugging Face `transformers` (BERT is not the only one transformer, you can experiment with others too) and `datasets` (again, you can experiment with other sentiment-analysis datasets). Install `sentence-splitter`.\n", |
|---|
| 29 | "\n", |
|---|
| 30 | "We will define manually the \"aspect words\" and find them (and similar words) in the sentences. For similarity, we use word embeddings.\n", |
|---|
| 31 | "\n" |
|---|
| 32 | ], |
|---|
| 33 | "metadata": { |
|---|
| 34 | "id": "WK14us8lcqWt" |
|---|
| 35 | } |
|---|
| 36 | }, |
|---|
| 37 | { |
|---|
| 38 | "cell_type": "markdown", |
|---|
| 39 | "source": [ |
|---|
| 40 | "## More details" |
|---|
| 41 | ], |
|---|
| 42 | "metadata": { |
|---|
| 43 | "id": "1xAYTR67fp73" |
|---|
| 44 | } |
|---|
| 45 | }, |
|---|
| 46 | { |
|---|
| 47 | "cell_type": "markdown", |
|---|
| 48 | "source": [ |
|---|
| 49 | "Aspect-based sentiment analysis makes sense in case of reviews, where the reviewer comments on several aspects (attributes) of the reviewed object. The sentiment polarity and intensity can be different for each aspect. Aspect-based sentiment analysis therefore provides more detailed insight into users' sentiments about the object than a simple 👍 or 👎 of the whole text.\n", |
|---|
| 50 | "\n", |
|---|
| 51 | "\n", |
|---|
| 52 | "The assumptions are:\n", |
|---|
| 53 | "* the text is structured in sentences\n", |
|---|
| 54 | "* the aspect is mentioned in the sentences together with the sentiment\n", |
|---|
| 55 | "* the sentiment analysis is performed in one domain, so the aspects are predefined (supervised)\n", |
|---|
| 56 | "\n", |
|---|
| 57 | "In order to have the aspects definitions more robust, we accept semantically similar words too. For this task, we use the word embeddings. For simplicity, we use the GloVe 50 (a small predefined word embeddings model). You can experiment with larger models that probably provide more accurate synonyms." |
|---|
| 58 | ], |
|---|
| 59 | "metadata": { |
|---|
| 60 | "id": "ve3pc7H3fs1n" |
|---|
| 61 | } |
|---|
| 62 | }, |
|---|
| 63 | { |
|---|
| 64 | "cell_type": "markdown", |
|---|
| 65 | "source": [ |
|---|
| 66 | "## " |
|---|
| 67 | ], |
|---|
| 68 | "metadata": { |
|---|
| 69 | "id": "8pC5yt2AftuQ" |
|---|
| 70 | } |
|---|
| 71 | }, |
|---|
| 72 | { |
|---|
| 73 | "cell_type": "code", |
|---|
| 74 | "source": [ |
|---|
| 75 | "!pip install transformers datasets" |
|---|
| 76 | ], |
|---|
| 77 | "metadata": { |
|---|
| 78 | "id": "LaYbzwWmy3KY" |
|---|
| 79 | }, |
|---|
| 80 | "execution_count": null, |
|---|
| 81 | "outputs": [] |
|---|
| 82 | }, |
|---|
| 83 | { |
|---|
| 84 | "cell_type": "code", |
|---|
| 85 | "source": [ |
|---|
| 86 | "!pip install sentence-splitter" |
|---|
| 87 | ], |
|---|
| 88 | "metadata": { |
|---|
| 89 | "id": "mdhXquZb_fXg" |
|---|
| 90 | }, |
|---|
| 91 | "execution_count": null, |
|---|
| 92 | "outputs": [] |
|---|
| 93 | }, |
|---|
| 94 | { |
|---|
| 95 | "cell_type": "code", |
|---|
| 96 | "source": [ |
|---|
| 97 | "!pip install gensim" |
|---|
| 98 | ], |
|---|
| 99 | "metadata": { |
|---|
| 100 | "id": "A6r_mvaIRZAT" |
|---|
| 101 | }, |
|---|
| 102 | "execution_count": null, |
|---|
| 103 | "outputs": [] |
|---|
| 104 | }, |
|---|
| 105 | { |
|---|
| 106 | "cell_type": "code", |
|---|
| 107 | "source": [ |
|---|
| 108 | "from datasets import load_dataset # load dataset stored in hugging face\n", |
|---|
| 109 | "from transformers import AutoModelForSequenceClassification # generalized task\n", |
|---|
| 110 | "from transformers import AutoTokenizer # the tokenizer has to be compatible with the model (so Auto* is a good choice)\n", |
|---|
| 111 | "from transformers import pipeline # pipeline for the whole task\n", |
|---|
| 112 | "import numpy as np\n", |
|---|
| 113 | "import gensim.downloader as api\n", |
|---|
| 114 | "from sentence_splitter import SentenceSplitter, split_text_into_sentences" |
|---|
| 115 | ], |
|---|
| 116 | "metadata": { |
|---|
| 117 | "id": "NK7gfY0IyyFe" |
|---|
| 118 | }, |
|---|
| 119 | "execution_count": null, |
|---|
| 120 | "outputs": [] |
|---|
| 121 | }, |
|---|
| 122 | { |
|---|
| 123 | "cell_type": "code", |
|---|
| 124 | "source": [ |
|---|
| 125 | "gv = api.load('glove-wiki-gigaword-50')" |
|---|
| 126 | ], |
|---|
| 127 | "metadata": { |
|---|
| 128 | "id": "8R47T18XRfpT" |
|---|
| 129 | }, |
|---|
| 130 | "execution_count": null, |
|---|
| 131 | "outputs": [] |
|---|
| 132 | }, |
|---|
| 133 | { |
|---|
| 134 | "cell_type": "markdown", |
|---|
| 135 | "source": [ |
|---|
| 136 | "# Get the data\n", |
|---|
| 137 | "In this notebook, we experiment with movie reviews. The IMDB dataset provides overall sentiment as a binary classification (positive/negative).\n", |
|---|
| 138 | "\n", |
|---|
| 139 | "**TASK1**: Observe the data, read some reviews, check the label. Do you agree with the label?" |
|---|
| 140 | ], |
|---|
| 141 | "metadata": { |
|---|
| 142 | "id": "7qVLN3rwfDcJ" |
|---|
| 143 | } |
|---|
| 144 | }, |
|---|
| 145 | { |
|---|
| 146 | "cell_type": "code", |
|---|
| 147 | "source": [ |
|---|
| 148 | "imdb = load_dataset(\"imdb\")" |
|---|
| 149 | ], |
|---|
| 150 | "metadata": { |
|---|
| 151 | "id": "bnOm7QUsRHud" |
|---|
| 152 | }, |
|---|
| 153 | "execution_count": null, |
|---|
| 154 | "outputs": [] |
|---|
| 155 | }, |
|---|
| 156 | { |
|---|
| 157 | "cell_type": "code", |
|---|
| 158 | "source": [ |
|---|
| 159 | "id_ = 7900\n", |
|---|
| 160 | "imdb['test'][id_]" |
|---|
| 161 | ], |
|---|
| 162 | "metadata": { |
|---|
| 163 | "id": "TF-ShPgeCbN3" |
|---|
| 164 | }, |
|---|
| 165 | "execution_count": null, |
|---|
| 166 | "outputs": [] |
|---|
| 167 | }, |
|---|
| 168 | { |
|---|
| 169 | "cell_type": "code", |
|---|
| 170 | "source": [ |
|---|
| 171 | "text=imdb['test'][id_]['text']" |
|---|
| 172 | ], |
|---|
| 173 | "metadata": { |
|---|
| 174 | "id": "BwqjfabNq_Ws" |
|---|
| 175 | }, |
|---|
| 176 | "execution_count": null, |
|---|
| 177 | "outputs": [] |
|---|
| 178 | }, |
|---|
| 179 | { |
|---|
| 180 | "cell_type": "code", |
|---|
| 181 | "source": [ |
|---|
| 182 | "splitter = SentenceSplitter(language='en')\n", |
|---|
| 183 | "splitter.split(text=text)" |
|---|
| 184 | ], |
|---|
| 185 | "metadata": { |
|---|
| 186 | "id": "0L37LN7Q1zi6" |
|---|
| 187 | }, |
|---|
| 188 | "execution_count": null, |
|---|
| 189 | "outputs": [] |
|---|
| 190 | }, |
|---|
| 191 | { |
|---|
| 192 | "cell_type": "markdown", |
|---|
| 193 | "source": [ |
|---|
| 194 | "# Pretrained Model for Sentiment Analysis\n", |
|---|
| 195 | "\n", |
|---|
| 196 | "We will experiment with one of the Sentiment Analysis models available in Hugging Face collection.\n" |
|---|
| 197 | ], |
|---|
| 198 | "metadata": { |
|---|
| 199 | "id": "E4rxw9NFgGzV" |
|---|
| 200 | } |
|---|
| 201 | }, |
|---|
| 202 | { |
|---|
| 203 | "cell_type": "markdown", |
|---|
| 204 | "source": [ |
|---|
| 205 | "## More details" |
|---|
| 206 | ], |
|---|
| 207 | "metadata": { |
|---|
| 208 | "id": "ivnqydtMgHpt" |
|---|
| 209 | } |
|---|
| 210 | }, |
|---|
| 211 | { |
|---|
| 212 | "cell_type": "markdown", |
|---|
| 213 | "source": [ |
|---|
| 214 | "\n", |
|---|
| 215 | "The model is available in https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment.\n", |
|---|
| 216 | "\n", |
|---|
| 217 | "Click on `Use in Transformers` to get the right class to `import`:\n", |
|---|
| 218 | "" |
|---|
| 219 | ], |
|---|
| 220 | "metadata": { |
|---|
| 221 | "id": "iIEneHNgflrR" |
|---|
| 222 | } |
|---|
| 223 | }, |
|---|
| 224 | { |
|---|
| 225 | "cell_type": "markdown", |
|---|
| 226 | "source": [ |
|---|
| 227 | "## " |
|---|
| 228 | ], |
|---|
| 229 | "metadata": { |
|---|
| 230 | "id": "6dO23AcBgKuc" |
|---|
| 231 | } |
|---|
| 232 | }, |
|---|
| 233 | { |
|---|
| 234 | "cell_type": "code", |
|---|
| 235 | "source": [ |
|---|
| 236 | "model_name = 'sbcBI/sentiment_analysis'\n", |
|---|
| 237 | "model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"" |
|---|
| 238 | ], |
|---|
| 239 | "metadata": { |
|---|
| 240 | "id": "XFc76_SORT6d" |
|---|
| 241 | }, |
|---|
| 242 | "execution_count": null, |
|---|
| 243 | "outputs": [] |
|---|
| 244 | }, |
|---|
| 245 | { |
|---|
| 246 | "cell_type": "code", |
|---|
| 247 | "source": [ |
|---|
| 248 | "tokenizer = AutoTokenizer.from_pretrained(model_name)\n", |
|---|
| 249 | "tokenizer_kwargs = {'padding':True,'truncation':True,'max_length':512}\n", |
|---|
| 250 | "model = AutoModelForSequenceClassification.from_pretrained(model_name)" |
|---|
| 251 | ], |
|---|
| 252 | "metadata": { |
|---|
| 253 | "id": "PkIekSozRWfG" |
|---|
| 254 | }, |
|---|
| 255 | "execution_count": null, |
|---|
| 256 | "outputs": [] |
|---|
| 257 | }, |
|---|
| 258 | { |
|---|
| 259 | "cell_type": "code", |
|---|
| 260 | "source": [ |
|---|
| 261 | "classifier = pipeline(\"sentiment-analysis\", model=model, tokenizer=tokenizer, **tokenizer_kwargs)" |
|---|
| 262 | ], |
|---|
| 263 | "metadata": { |
|---|
| 264 | "id": "0Ke-Cwwe2jpo" |
|---|
| 265 | }, |
|---|
| 266 | "execution_count": null, |
|---|
| 267 | "outputs": [] |
|---|
| 268 | }, |
|---|
| 269 | { |
|---|
| 270 | "cell_type": "code", |
|---|
| 271 | "source": [ |
|---|
| 272 | "classifier(text)" |
|---|
| 273 | ], |
|---|
| 274 | "metadata": { |
|---|
| 275 | "id": "4FQysbn-3dYR" |
|---|
| 276 | }, |
|---|
| 277 | "execution_count": null, |
|---|
| 278 | "outputs": [] |
|---|
| 279 | }, |
|---|
| 280 | { |
|---|
| 281 | "cell_type": "markdown", |
|---|
| 282 | "source": [ |
|---|
| 283 | "# Look for Aspects\n", |
|---|
| 284 | "First, we define manually the aspects relevant to the dataset.\n", |
|---|
| 285 | "\n", |
|---|
| 286 | "Next, we use the BERT model to detect sentiment of each sentence. At the same time, if the aspect is mentioned in the sentence, we assume it is the sentiment towards the aspect.\n", |
|---|
| 287 | "\n", |
|---|
| 288 | "**TASK2**: Go back to observed data and find some aspects the reviewers evaluate." |
|---|
| 289 | ], |
|---|
| 290 | "metadata": { |
|---|
| 291 | "id": "7dXOriSNgtmk" |
|---|
| 292 | } |
|---|
| 293 | }, |
|---|
| 294 | { |
|---|
| 295 | "cell_type": "code", |
|---|
| 296 | "source": [ |
|---|
| 297 | "aspects = {'actor': ['actor', 'actress'],\n", |
|---|
| 298 | " 'action': ['action']}" |
|---|
| 299 | ], |
|---|
| 300 | "metadata": { |
|---|
| 301 | "id": "Pb0ltpwhR0a-" |
|---|
| 302 | }, |
|---|
| 303 | "execution_count": null, |
|---|
| 304 | "outputs": [] |
|---|
| 305 | }, |
|---|
| 306 | { |
|---|
| 307 | "cell_type": "code", |
|---|
| 308 | "source": [ |
|---|
| 309 | "def cosine(A, B):\n", |
|---|
| 310 | " return np.dot(A,B)/(np.linalg.norm(A)*np.linalg.norm(B))" |
|---|
| 311 | ], |
|---|
| 312 | "metadata": { |
|---|
| 313 | "id": "hPVnja88FH82" |
|---|
| 314 | }, |
|---|
| 315 | "execution_count": null, |
|---|
| 316 | "outputs": [] |
|---|
| 317 | }, |
|---|
| 318 | { |
|---|
| 319 | "cell_type": "code", |
|---|
| 320 | "source": [ |
|---|
| 321 | "predicted_label = classifier(text)[0]['label']\n", |
|---|
| 322 | "print('Overall classification', predicted_label, imdb['test'][id_]['label'])\n", |
|---|
| 323 | "if (imdb['test'][id_]['label']==0 and predicted_label in ['1 star', '2 stars', '3 stars']) or (imdb['test'][id_]['label']==1 and predicted_label in ['4 stars', '5 stars']):\n", |
|---|
| 324 | " print('True')\n", |
|---|
| 325 | "else:\n", |
|---|
| 326 | " print('False')" |
|---|
| 327 | ], |
|---|
| 328 | "metadata": { |
|---|
| 329 | "id": "79VWN6YitB0M" |
|---|
| 330 | }, |
|---|
| 331 | "execution_count": null, |
|---|
| 332 | "outputs": [] |
|---|
| 333 | }, |
|---|
| 334 | { |
|---|
| 335 | "cell_type": "code", |
|---|
| 336 | "source": [ |
|---|
| 337 | "threshold = 0.8\n", |
|---|
| 338 | "splitter = SentenceSplitter(language='en')\n", |
|---|
| 339 | "absa = {}\n", |
|---|
| 340 | "for sentence in splitter.split(text=text):\n", |
|---|
| 341 | "# print(sentence)\n", |
|---|
| 342 | " for token in tokenizer.tokenize(sentence):\n", |
|---|
| 343 | " for aspect_name, aspect_values in aspects.items():\n", |
|---|
| 344 | " for aspect in aspect_values:\n", |
|---|
| 345 | " if token in gv and aspect in gv:\n", |
|---|
| 346 | " similarity = cosine(gv[aspect], gv[token])\n", |
|---|
| 347 | " if similarity>threshold:\n", |
|---|
| 348 | " if aspect_name not in absa.keys():\n", |
|---|
| 349 | " absa[aspect_name] = []\n", |
|---|
| 350 | " if token not in [t['token'] for t in absa[aspect_name]]:\n", |
|---|
| 351 | " absa[aspect_name].append({'token': token, 'label': classifier(sentence)})\n", |
|---|
| 352 | " #print(classifier(sentence))\n", |
|---|
| 353 | "print('Result')\n", |
|---|
| 354 | "absa" |
|---|
| 355 | ], |
|---|
| 356 | "metadata": { |
|---|
| 357 | "id": "LYxGTZmsCHMm" |
|---|
| 358 | }, |
|---|
| 359 | "execution_count": null, |
|---|
| 360 | "outputs": [] |
|---|
| 361 | }, |
|---|
| 362 | { |
|---|
| 363 | "cell_type": "markdown", |
|---|
| 364 | "source": [ |
|---|
| 365 | "If you uncomment the texts, you will see sentiment classification for each sentence." |
|---|
| 366 | ], |
|---|
| 367 | "metadata": { |
|---|
| 368 | "id": "OwaIpowDgcJA" |
|---|
| 369 | } |
|---|
| 370 | }, |
|---|
| 371 | { |
|---|
| 372 | "cell_type": "markdown", |
|---|
| 373 | "source": [ |
|---|
| 374 | "# Simple but Efficient?\n", |
|---|
| 375 | "\n", |
|---|
| 376 | "So far, we did not measure whether this simple approach is good. Let's check on some aspect-based annotated data.\n", |
|---|
| 377 | "\n", |
|---|
| 378 | "**TASK3**: Use part of the TripAdvisor data to evaluate whether our approach can detect the aspect and their polarity in the data.\n", |
|---|
| 379 | "\n", |
|---|
| 380 | "**UPDATED 2023-09-19 based on feedback in the lecture**\n", |
|---|
| 381 | "\n", |
|---|
| 382 | "The TripAdvisor dataset contains *ground truth* values not only for the overall sentiment but also for aspects of the hotel. The aspects are Service, Cleanliness, Value, Sleep Quality, Rooms, Location and maybe some others. The Overall is the overall evaluation.\n", |
|---|
| 383 | "\n", |
|---|
| 384 | "The TripAdvisor review example is below:\n", |
|---|
| 385 | "\n", |
|---|
| 386 | "```json\n", |
|---|
| 387 | "\"Ratings\": {\n", |
|---|
| 388 | " \"Service\": \"5\",\n", |
|---|
| 389 | " \"Cleanliness\": \"5\",\n", |
|---|
| 390 | " \"Overall\": \"5.0\",\n", |
|---|
| 391 | " \"Value\": \"4\",\n", |
|---|
| 392 | " \"Sleep Quality\": \"3\",\n", |
|---|
| 393 | " \"Rooms\": \"3\",\n", |
|---|
| 394 | " \"Location\": \"3\"\n", |
|---|
| 395 | " },\n", |
|---|
| 396 | " \"AuthorLocation\": \"Surrey, Canada\",\n", |
|---|
| 397 | " \"Title\": \"“Service was Excellent”\",\n", |
|---|
| 398 | " \"Author\": \"susan h\",\n", |
|---|
| 399 | " \"ReviewID\": \"UR123992579\",\n", |
|---|
| 400 | " \"Content\": \"This was a very clean hotel. Not fancy but served the purpose. The service from the employees however was FANTASTIC! I would recommend to anyone not looking for swanky but good clean room at a very reasonable rate.\",\n", |
|---|
| 401 | " \"Date\": \"February 1, 2012\"\n", |
|---|
| 402 | "```\n", |
|---|
| 403 | "\n", |
|---|
| 404 | "1. download the TripAdvisor dataset (only the JSON part) from https://www.cs.virginia.edu/~hw5x/Data/LARA/TripAdvisor/, bunzip (`bunzip TripAdvisorJson.tar.bz2`), untar (`tar -xvf TripAdvisorJson.tar`)\n", |
|---|
| 405 | "1. pick **one json** file (20-60 ratings)\n", |
|---|
| 406 | "1. find aspect words for the domain and put them in the `aspects` dictionary (at least five from the `Ratings` except of the `Overall`). Your `aspects` dictionary will look like:\n", |
|---|
| 407 | "```python\n", |
|---|
| 408 | "aspects = {'Location': ['place', ...], 'Value': ['money', ...], ...}\n", |
|---|
| 409 | "```\n", |
|---|
| 410 | "1. process the JSON data using our approach - you will obtain the same aspects as in the TripAdvisor dataset\n", |
|---|
| 411 | "1. write down how many times the aspect was found (e.g. from 25 reviews, the aspect Location was found 14 times)\n", |
|---|
| 412 | "1. find in how many cases there was an agreement (exact match), optionally, check whether the sentiment was detected correctly (instead of stars, use 1-3 for negative and 4-5 for positive). For example, in 14 cases where the Location aspect was found, it was predicted correctly in 8 cases.\n", |
|---|
| 413 | "1. write the results for each aspect in the text file\n", |
|---|
| 414 | "1. iterate to find good aspect words and suitable word embedding threshold\n", |
|---|
| 415 | "1. submit the best result in the homework vault\n", |
|---|
| 416 | "\n", |
|---|
| 417 | "Your submission will look like:\n", |
|---|
| 418 | "```txt\n", |
|---|
| 419 | "selected file: 123567.json (28 reviews)\n", |
|---|
| 420 | "\n", |
|---|
| 421 | "aspects = {...}\n", |
|---|
| 422 | "\n", |
|---|
| 423 | "aspects found in the reviews:\n", |
|---|
| 424 | "Location 15/28\n", |
|---|
| 425 | "Sleep Quality 3/28\n", |
|---|
| 426 | "...\n", |
|---|
| 427 | "\n", |
|---|
| 428 | "exact match:\n", |
|---|
| 429 | "Location 13/15\n", |
|---|
| 430 | "Sleep Quality 0/3\n", |
|---|
| 431 | "\n", |
|---|
| 432 | "However, I observed that ...\n", |
|---|
| 433 | "\n", |
|---|
| 434 | "```\n" |
|---|
| 435 | ], |
|---|
| 436 | "metadata": { |
|---|
| 437 | "id": "QST3r5tbh5so" |
|---|
| 438 | } |
|---|
| 439 | }, |
|---|
| 440 | { |
|---|
| 441 | "cell_type": "markdown", |
|---|
| 442 | "source": [ |
|---|
| 443 | "## Hints\n" |
|---|
| 444 | ], |
|---|
| 445 | "metadata": { |
|---|
| 446 | "id": "x_qd8O0fgs8M" |
|---|
| 447 | } |
|---|
| 448 | }, |
|---|
| 449 | { |
|---|
| 450 | "cell_type": "markdown", |
|---|
| 451 | "source": [ |
|---|
| 452 | "* Modify the loop over sentences, tokens, and aspects in a function. Calculate rating for the aspect as the average of star evaluations.\n", |
|---|
| 453 | "* To convert the stars into the TripAdvisor rating, simply pick first character of the model prediction and convert it to `int` or `float`.\n", |
|---|
| 454 | "* If you increase the number of \"aspect words\", you increase the recall but the precision might drop." |
|---|
| 455 | ], |
|---|
| 456 | "metadata": { |
|---|
| 457 | "id": "wdsr0v3eg-dQ" |
|---|
| 458 | } |
|---|
| 459 | }, |
|---|
| 460 | { |
|---|
| 461 | "cell_type": "markdown", |
|---|
| 462 | "source": [ |
|---|
| 463 | "## " |
|---|
| 464 | ], |
|---|
| 465 | "metadata": { |
|---|
| 466 | "id": "PXdU9O5fg29O" |
|---|
| 467 | } |
|---|
| 468 | } |
|---|
| 469 | ] |
|---|
| 470 | } |
|---|
