private/NlpInPracticeCourse/InformationExtraction: IA161_Hypernym_Extraction.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Hypernym Extraction\n",
        "Hypernym extraction is crucial in lexicography (a good start of a dictionary definition is the hypernym) and ontology engineering (hypernym-hyponym or subclass-superclass is the backbone relation is ontologies).\n",
        "\n",
        "## Get the data\n",
        "\n",
        "Download input data using wget.\n",
        "Check the nature of the data by listing first n lines (by default n=10)."
      ],
      "metadata": {
        "id": "LCYhjKC9TosE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget http://nlp.fi.muni.cz/trac/research/raw-attachment/wiki/private/NlpInPracticeCourse/RelationExtraction/input.txt"
      ],
      "metadata": {
        "id": "_zIYBky4TzlW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!head input.txt"
      ],
      "metadata": {
        "id": "thqNaB3zEcq6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Tools\n",
        "\n",
        "### Tokenizer and PoS tagger\n",
        "\n",
        "We use NLTK standard tokenizer to split the texts into words. And then NLTK Part-of-Speech tagger to get PoS information."
      ],
      "metadata": {
        "id": "7U_pqaF6UOe_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "nltk.download('punkt')\n",
        "nltk.download('averaged_perceptron_tagger')"
      ],
      "metadata": {
        "id": "sHUnUjjZUoCt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Extract hypernyms\n",
        "\n",
        "Our script reads the lines from input.txt (in the form: word|definition) and calls *find_hyper* method to detect hypernym from the definition. The detection is quite naive, it takes first word tagged as noun (NN). You should extend and modify *find_hyper* method to produce better results."
      ],
      "metadata": {
        "id": "Bcfjj8zjUt99"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def find_hyper(word, text):\n",
        "    tokenized = word_tokenize(text)\n",
        "    pos_tagged = nltk.pos_tag(tokenized)\n",
        "    # find first noun\n",
        "    nouns = [tup for tup in pos_tagged if tup[1] == 'NN']\n",
        "    if len(nouns) > 0:\n",
        "        return (word, nouns[0][0])\n",
        "    return (word, u'')\n",
        "\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "\tinput_data = 'input.txt'\n",
        "\tfor line in open(input_data, 'r'):\n",
        "\t\tif line.strip() != '':\n",
        "\t\t\tline_split = line.strip().split('|')\n",
        "\t\t\thyper = find_hyper(line_split[0], line_split[1])\n",
        "\t\t\tprint(\"%s = %s\" % hyper)\n"
      ],
      "metadata": {
        "id": "teuFO7p_VE0y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Evaluate\n",
        "For evaluation, select some straighforward metric such as accuracy. Calculate and publish the performance of the naive approach, then re-calculate and publish performance for improved `find_hyper` function. For evaluation, use the `gold_en.txt`."
      ],
      "metadata": {
        "id": "jcKqLv7mY0X-"
      }
    }
  ]
}