Introduction to Named Entity Recognition with Python
Introduction
This repository contains Jupyter notebooks used for teaching ‘Text-mining with Python: Named Entity Recognition (NER)’,
a course in the annual Cambridge Digital Humanities (CDH) Cultural Heritage Data School.
The notebooks are designed to be worked on as self-paced materials in a ‘flipped classroom’ approach. They
are also written as stand-alone notebooks for anyone to follow and use as they wish.
The aim is to teach basic NER techniques to a wide audience, and the material suitable both for those people who:
- Have some background in Python or
- Just want to learn about the concepts without programming.
Please note that the notebooks are designed to be run as a teaching aid, not as a serious text analysis tool.
Contents
Using the example of some nineteenth-century letters of science, these notebooks introduce how to:
- Automatically recognise and visualise named entities using machine learning;
- Train machine learning models for improving results;
- Link named entities to existing knowledge bases or authorities.
Recommended Pathways
For non-coders, I recommend you start with the notebook
2-named-entity-recognition-of-henslow-data and skip over the
notebook 4-updating-the-model-on-henslow-data.
If you’re a Python beginner, I recommend that you work through the first notebook
0-introduction-to-python-and-text.ipynb. It contains a brief refresher
of the basic Python you need to understand the NER examples, but I don’t expect it will be enough to teach you Python
from scratch.
For everyone else, I recommend you start with notebook
1-basic-text-mining-concepts.ipynb and work through the rest in order.
4-updating-the-model-on-henslow-data.ipynb is the most advanced Python in the course and is intended as a deep dive for those with a further interest in
working with spaCy. If you don’t understand everything here yet, feel free to run through it quickly or skip it.
Data and Licensing
Originally, these notebooks were delivered using data from the
Darwin Correspondence Project, which we had permission to use within the
context of the Data School. Unfortunately, the CC-BY-NC-ND license
under which the letters are licensed from Cambridge University Press allows for distribution of the letters, but not
the creation of derivatives, which meant that the notebooks could not be published. I have re-written the notebooks
using data from the Henslow Correspondence Project
(HCP) instead, which is licensed more permissively with CC-BY-NC.
Content Warning
The HCP letters were written during the period of British imperialism, therefore some of the correspondence contains
content we now find offensive, for example, letters_138.xml contains a racist description. These notebooks do not
contain or discuss any of this material, but please be aware you may come across it if you browse through the letters
independently.
Quick Start: Launch Notebooks Online
For a Quick Look: Run on Binder
If you just want to play quickly with the notebooks and see what they show, click on the “launch binder” button.
Binder will launch a virtual environment in your browser where you can open and run the notebooks without installing
anything.
Limitations of Binder:
- Some cells in the notebooks may use more memory than Binder allows, causing the notebook’s kernel to crash. After it
has restarted, try modifying the code to process fewer documents.
- Binder may shut down after about 10 minutes of inactivity e.g. if you don’t keep the window open. You can simply
open a new Binder to start again.
- Binder will not save any changes you make to the notebooks.
Run in the Cloud without any Installation: Run on Deepnote
To run and keep a copy of the notebooks for yourself, click on the “Launch in Deepnote” button. Deepnote will create
a project based on this repository automatically and run in the cloud, so you don’t have to install anything on your
local computer.
After the project has started, go to the buttons on the left-hand side and click on the Environment button (1) to open
the Environment tab. In the Environment tab, click (2) new Docker image.
A dialog will pop up: type in camdl/deepnote:python3.7 and then click Add environment and apply.
After a minute or two, Deepnote will start the new machine and run the install steps. Then the notebooks are ready to
use. Click on the Folder icon to open the list of notebooks.
Limitations of Deepnote:
- Deepnote requires you to sign up for an account.
- Deepnote has a (generous) limit on the number of free hours you can use each month.
- On the free tier, the notebooks will likely run slower than on your own computer.
Local Installation
Please note these instructions are suitable if you already have Python installed in some way. If you have never
installed Python yourself on your computer before, I recommend this guide:
Python 3 Installation & Setup Guide.
Important!
The notebooks should run on any of the following versions of Python:
The notebooks have not yet been updated to run on Python 3.9 and above. If you try to run them on
Python 3.9 or above, you will get errors when you install the dependencies.
Download or Clone the Files
Click the green “Code” button to the top-right of this page.
If you have never used git version control before I recommend you simply download the notebooks with the
“Download ZIP” option. In most operating systems this will automatically unzip it back into individual files. Move
the folder to somewhere you want to keep it, such as “My Documents”.
If you have used git before, then you can clone the repo with this command:
git clone https://github.com/mchesterkadwell/named-entity-recognition.git
Vanilla/Plain Python
If you installed Python from python.org (or from the Windows Store) follow these
instructions.
If you are using PyCharm or another IDE with which you are already familiar, of course, do what you
normally do instead to create a virtual environment and install dependencies.
Mac & Linux: Setting up the Notebook Server for the First Time
Open a Terminal and change directory into the notebooks folder by typing something like this:
cd path/to/notebooks
where path/to/notebooks is the filepath to wherever you’ve put the notebooks folder.
Then create a new virtual environment:
python3 -m venv env
Activate the virtual environment:
source env/bin/activate
Then install all the dependencies:
pip install -r requirements.txt
This should initiate a big list of downloads and will take a while to finish. Please be patient.
Finally, to start the notebook server type:
jupyter notebook
When you are finished with the notebook, press ctrl+c to stop the notebook server. Then type:
deactivate
You can close the Terminal window.
Mac & Linux: Starting the Notebook Server Again
Open a terminal and type something like this (pressing return between each line):
cd path/to/notebooks
source env/bin/activate
jupyter notebook
When you are finished with the notebook, press ctrl+c to stop the notebook server. Then type:
deactivate
You can close the Terminal window.
Windows: Setting up the Notebook Server for the First Time
Open a Command Prompt and change directory into the notebooks folder by typing something like this:
cd path\to\notebooks
where path\to\notebooks is the filepath to wherever you’ve put the notebooks folder.
Then create a new virtual environment:
python -m venv env
Activate the virtual environment:
env\Scripts\activate.bat
Then install all the dependencies:
pip install -r requirements.txt
This should initiate a big list of downloads and will take a while to finish. Please be patient.
Finally, to start the notebook server type:
jupyter notebook
When you are finished with the notebook, press ctrl+c to stop the notebook server. Then type:
deactivate
You can close the Command Prompt window.
Windows: Starting the Notebook Server Again
Open a Command Prompt and type something like this (pressing return between each line):
cd path\to\notebooks
env\Scripts\activate.bat
jupyter notebook
When you are finished with the notebook, press ctrl+c to stop the notebook server. Then type:
deactivate
You can close the Command Prompt window.
Anaconda
If you installed Python with Anaconda from Anaconda.com follow these
instructions.
Setting up the Notebook Server for the First Time
Open Anaconda Navigator. In Anaconda Navigator > Environments click on the ‘Create’ button in the bottom left of the
Environments list.
Type a name e.g. ‘data-school-ner’, make sure that ‘Python’ is checked
and under the dropdown pick ‘3.7’ or ‘3.8’. Make sure that ‘R’ is left unchecked.
Then click the ‘Create’ button.
It will take a few seconds to set up…
Then in Anaconda Navigator > Environments make sure you have selected your new environment.
On the right of the environment name is a small green play arrow. Click on it and pick ‘Open Terminal’ from the
dropdown.
In the Terminal that opens type the following, and press return:
conda install pip
If you do not already have pip installed, it will install it. Otherwise it will give a message:
# All requested packages already installed.
Then change directory to wherever you saved the notebooks folder by typing something like:
cd \path\to\notebooks
where path\to\notebooks is the filepath to wherever you’ve put the notebooks folder.
If you are on Mac or Linux, make sure to use forward slashes in the filepath instead e.g. path/to/notebooks
Then install all the dependencies by typing:
pip install -r requirements.txt
This should initiate a big list of downloads and will take a while to finish. Please be patient.
Finally, to launch the Jupyter notebook server type:
jupyter notebook
It should automatically open a browser window with the notebook listing in it, a bit like this:
If not, you can copy and paste one of the URLs in the Terminal window into your browser e.g.
http://localhost:8888/?token=ddb27d2a1a6cb29a3483c24d6ff9f7263eb9676f02d71075
(This example will not work on your machine, as the token is unique every time!)
When you are finished with the notebook, press ctrl+c to stop the notebook server.
You can close the Terminal window.
Starting the Notebook Server Again
Next time you want to start the notebook server:
In Anaconda Navigator > Environments make sure you have selected your new environment e.g. ‘data-school-ner’.
On the right of the environment name is a small green play arrow. Click on it and pick ‘Open Terminal’ from the
dropdown.
To launch the Jupyter notebook server type:
jupyter notebook
When you are finished with the notebook, press ctrl+c to stop the notebook server. You can close the Terminal
window.
评论