SkyID
/
deepke
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Update README

This commit is contained in:
xxupiano 2021-11-30 14:26:22 +08:00
parent c58b226929
commit 1e00e429b4
2 changed files with 214 additions and 99 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

149
README.md
View File

@ -18,22 +18,23 @@
<br> <br>
<h2 align="center"> <h1 align="center">
<p>A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base Population</p> <p>A Deep Learning Based Knowledge Extraction Toolkit<br>for Knowledge Base Population</p>
</h2> </h1>
DeepKE is a knowledge extraction toolkit supporting **low-resource** and **document-level** scenarios. It provides three functions based on **PyTorch**, including **Named Entity Recognition**, **Relation Extraciton** and **Attribute Extraction**. DeepKE is a knowledge extraction toolkit supporting **low-resource** and **document-level** scenarios. It provides three functions based on **PyTorch**, including **Named Entity Recognition**, **Relation Extraciton** and **Attribute Extraction**.
<br> <br>
## Prediction # Prediction
There is a demonstration of prediction.<br> There is a demonstration of prediction.<br>
<img src="pics/demo.gif" width="636" height="494" align=center> <img src="pics/demo.gif" width="636" height="494" align=center>
<br> <br>
## Model Framework # Model Framework
<h3 align="center"> <h3 align="center">
<img src="pics/architectures.png"> <img src="pics/architectures.png">
@ -42,23 +43,25 @@ There is a demonstration of prediction.<br>
Figure 1: The framework of DeepKE Figure 1: The framework of DeepKE
</p> </p>
- DeepKE contains three modules for **named entity recognition**, **relation extraction** and **attribute extraction**, the three tasks respectively. - DeepKE contains a unified framework for **named entity recognition**, **relation extraction** and **attribute extraction**, the three knowledge extraction functions.
- Each module has its own submodules. For example, there are **standard**, **document-level** and **few-shot** submodules in the attribute extraction modular. - Each task can be implemented in different scenarios. For example, we can achieve relation extraction in **standard**, **low-resource (few-shot)** and **document-level** settings.
- Each submodule compose of three parts: a **collection of tools**, which can function as tokenizer, dataloader, preprocessor and the like, a **encoder** and a part for **training and prediction** - Each application scenario comprises of three components: **Data** including Tokenizer, Preprocessor and Loader, **Model** including Module, Encoder and Forwarder, **Core** including Training, Evaluation and Prediction.
<br> <br>
## Quickstart # Quickstart
*DeepKE* is supported `pip install deepke`. Take the fully supervised attribute extraction for example. *DeepKE* is supported `pip install deepke`. Take the fully supervised relation extraction for example. <br>(Please star✨ and fork :memo: !!!)
**Step1** Download basic codes `git clone https://github.com/zjunlp/DeepKE.git ` (Please star✨ and fork :memo:) **Step1** Download the basic codes
**Step2** Create a virtual environment using`Anaconda` and enter it. ```bash
git clone https://github.com/zjunlp/DeepKE.git
We also provide dockerfile source code, you can create your own image, which is located in the docker folder. ```
**Step2** Create a virtual environment using `Anaconda` and enter it.<br>
We also provide dockerfile source code, you can create your own image, which is located in the `docker` folder.
```bash ```bash
conda create -n deepke python=3.8 conda create -n deepke python=3.8
@ -86,21 +89,29 @@ conda activate deepke
cd DeepKE/example/re/standard cd DeepKE/example/re/standard
``` ```
**Step4** Training (Parameters for training can be changed in the `conf` folder) **Step4** Download the dataset
```bash
wget 120.27.214.45/Data/re/standard/data.tar.gz
tar -xzvf data.tar.gz
```
**Step5** Training (Parameters for training can be changed in the `conf` folder)
```bash ```bash
python run.py python run.py
``` ```
**Step5** Prediction (Parameters for prediction can be changed in the `conf` folder) **Step6** Prediction (Parameters for prediction can be changed in the `conf` folder)
```bash ```bash
python predict.py python predict.py
``` ```
<br>
# Requirements
### Requirements
> python == 3.8 > python == 3.8
@ -117,9 +128,11 @@ python predict.py
- opt-einsum==3.3.0 - opt-einsum==3.3.0
- ujson - ujson
### Introduction of Three Functions <br>
#### 1. Named Entity Recognition # Introduction of Three Functions
## 1. Named Entity Recognition
- Named entity recognition seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, organizations, etc. - Named entity recognition seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, organizations, etc.
@ -134,9 +147,17 @@ python predict.py
- Read the detailed process in specific README - Read the detailed process in specific README
- **[STANDARD (Fully Supervised)](https://github.com/zjunlp/DeepKE/tree/main/example/ner/standard)** - **[STANDARD (Fully Supervised)](https://github.com/zjunlp/DeepKE/tree/main/example/ner/standard)**
**Step1** Enter `DeepKE/example/ner/standard`. The dataset and parameters can be customized in the `data` folder and `conf` folder respectively.<br> **Step1** Enter `DeepKE/example/ner/standard`. Download the dataset.
**Step2** Training ```bash
wget 120.27.214.45/Data/ner/standard/data.tar.gz
tar -xzvf data.tar.gz
```
**Step2** Training<br>
The dataset and parameters can be customized in the `data` folder and `conf` folder respectively.
```bash ```bash
python run.py python run.py
@ -150,15 +171,23 @@ python predict.py
- **[FEW-SHOT](https://github.com/zjunlp/DeepKE/tree/main/example/ner/few-shot)** - **[FEW-SHOT](https://github.com/zjunlp/DeepKE/tree/main/example/ner/few-shot)**
**Step1** Enter `DeepKE/example/ner/few-shot`. The directory where the model is loaded and saved and the configuration parameters can be cusomized in the `conf` folder.<br> **Step1** Enter `DeepKE/example/ner/few-shot`. Download the dataset.
**Step2** Training with default `CoNLL-2003` dataset. ```bash
wget 120.27.214.45/Data/ner/few_shot/data.tar.gz
tar -xzvf data.tar.gz
```
**Step2** Training in the low-resouce setting <br>
The directory where the model is loaded and saved and the configuration parameters can be cusomized in the `conf` folder.
```bash ```bash
python run.py +train=few_shot python run.py +train=few_shot
``` ```
Users can modify `load_path` in `conf/train/few_shot.yaml` with the use of existing loaded model.<br> Users can modify `load_path` in `conf/train/few_shot.yaml` to use existing loaded model.<br>
**Step3** Add `- predict` to `conf/config.yaml`, modify `loda_path` as the model path and `write_path` as the path where the predicted results are saved in `conf/predict.yaml`, and then run `python predict.py` **Step3** Add `- predict` to `conf/config.yaml`, modify `loda_path` as the model path and `write_path` as the path where the predicted results are saved in `conf/predict.yaml`, and then run `python predict.py`
@ -166,7 +195,7 @@ python predict.py
python predict.py python predict.py
``` ```
#### 2. Relation Extraction ## 2. Relation Extraction
- Relationship extraction is the task of extracting semantic relations between entities from a unstructured text. - Relationship extraction is the task of extracting semantic relations between entities from a unstructured text.
@ -180,11 +209,19 @@ python predict.py
- Read the detailed process in specific README - Read the detailed process in specific README
- **[STANDARD (Fully Supervised)](https://github.com/zjunlp/deepke/blob/test_new_deepke/example/re/standard)** - **[STANDARD (Fully Supervised)](https://github.com/zjunlp/DeepKE/tree/main/example/re/standard)**
**Step1** Enter the `DeepKE/example/re/standard` folder. The dataset and parameters can be customized in the `data` folder and `conf` folder respectively.<br> **Step1** Enter the `DeepKE/example/re/standard` folder. Download the dataset.
**Step2** Training ```bash
wget 120.27.214.45/Data/re/standard/data.tar.gz
tar -xzvf data.tar.gz
```
**Step2** Training<br>
The dataset and parameters can be customized in the `data` folder and `conf` folder respectively.
```bash ```bash
python run.py python run.py
@ -196,11 +233,20 @@ python predict.py
python predict.py python predict.py
``` ```
- **[FEW-SHOT](https://github.com/zjunlp/deepke/blob/test_new_deepke/example/re/few-shot)** - **[FEW-SHOT](https://github.com/zjunlp/DeepKE/tree/main/example/re/few-shot)**
**Step1** Enter `DeepKE/example/re/few-shot`. The dataset and parameters can be customized in the `data` folder and `conf` folder respectively.<br> **Step1** Enter `DeepKE/example/re/few-shot`. Download the dataset.
**Step 2** Training. Start with the model trained last time: modify `train_from_saved_model` in `conf/train.yaml`as the path where the model trained last time was saved. And the path saving logs generated in training can be customized by `log_dir`. <br> ```bash
wget 120.27.214.45/Data/re/few_shot/data.tar.gz
tar -xzvf data.tar.gz
```
**Step 2** Training<br>
- The dataset and parameters can be customized in the `data` folder and `conf` folder respectively.
- Start with the model trained last time: modify `train_from_saved_model` in `conf/train.yaml`as the path where the model trained last time was saved. And the path saving logs generated in training can be customized by `log_dir`.
```bash ```bash
python run.py python run.py
@ -212,13 +258,20 @@ python predict.py
python predict.py python predict.py
``` ```
- **[DOCUMENT](https://github.com/zjunlp/deepke/blob/test_new_deepke/example/re/document)**<br> - **[DOCUMENT](https://github.com/zjunlp/DeepKE/tree/main/example/re/document)**<br>
Download the model `train_distant.json` from [*Google Drive*](https://drive.google.com/drive/folders/1c5-0YwnoJx8NS6CV2f-NoTHR__BdkNqw) to `data/`. **Step1** Enter `DeepKE/example/re/document`. Download the dataset.
**Step1** Enter `DeepKE/example/re/document`. The dataset and parameters can be customized in the `data` folder and `conf` folder respectively.<br> ```bash
wget 120.27.214.45/Data/re/document/data.tar.gz
**Step2** Training. Start with the model trained last time: modify `train_from_saved_model` in `conf/train.yaml`as the path where the model trained last time was saved. And the path saving logs generated in training can be customized by `log_dir`. tar -xzvf data.tar.gz
```
**Step2** Training<br>
- The dataset and parameters can be customized in the `data` folder and `conf` folder respectively.
- Start with the model trained last time: modify `train_from_saved_model` in `conf/train.yaml`as the path where the model trained last time was saved. And the path saving logs generated in training can be customized by `log_dir`.
```bash ```bash
python run.py python run.py
@ -230,7 +283,7 @@ python predict.py
python predict.py python predict.py
``` ```
#### 3. Attribute Extraction ## 3. Attribute Extraction
- Attribute extraction is to extract attributes for entities in a unstructed text. - Attribute extraction is to extract attributes for entities in a unstructed text.
@ -243,11 +296,19 @@ python predict.py
| 2014年10月1日许鞍华执导的电影《黄金时代》上映 | 上映时间 | 黄金时代 | 19 | 2014年10月1日 | 0 | | 2014年10月1日许鞍华执导的电影《黄金时代》上映 | 上映时间 | 黄金时代 | 19 | 2014年10月1日 | 0 |
- Read the detailed process in specific README - Read the detailed process in specific README
- **[STANDARD (Fully Supervised)](https://github.com/zjunlp/deepke/blob/test_new_deepke/example/ae/standard)** - **[STANDARD (Fully Supervised)](https://github.com/zjunlp/DeepKE/tree/main/example/ae/standard)**
**Step1** Enter the `DeepKE/example/ae/standard` folder. The dataset and parameters can be customized in the `data` folder and `conf` folder respectively.<br> **Step1** Enter the `DeepKE/example/ae/standard` folder. Download the dataset.
**Step2** Training ```bash
wget 120.27.214.45/Data/ae/standard/data.tar.gz
tar -xzvf data.tar.gz
```
**Step2** Training<br>
The dataset and parameters can be customized in the `data` folder and `conf` folder respectively.
```bash ```bash
python run.py python run.py
@ -259,9 +320,9 @@ python predict.py
python predict.py python predict.py
``` ```
<br>
# Notebook Tutorial
## Notebook Tutorial
This toolkit provides many `Jupyter Notebook` and `Google Colab` tutorials. Users can study *DeepKE* with them. This toolkit provides many `Jupyter Notebook` and `Google Colab` tutorials. Users can study *DeepKE* with them.
@ -297,7 +358,7 @@ This toolkit provides many `Jupyter Notebook` and `Google Colab` tutorials. User
<br> <br>
## Tips # Tips
1. Using nearest mirror, like [THU](https://mirrors.tuna.tsinghua.edu.cn/help/anaconda/) in China, will speed up the installation of *Anaconda*. 1. Using nearest mirror, like [THU](https://mirrors.tuna.tsinghua.edu.cn/help/anaconda/) in China, will speed up the installation of *Anaconda*.
2. Using nearest mirror, like [aliyun](http://mirrors.aliyun.com/pypi/simple/) in China, will speed up `pip install XXX`. 2. Using nearest mirror, like [aliyun](http://mirrors.aliyun.com/pypi/simple/) in China, will speed up `pip install XXX`.
@ -307,7 +368,7 @@ This toolkit provides many `Jupyter Notebook` and `Google Colab` tutorials. User
<br> <br>
## Developers # Developers
Zhejiang University: Ningyu Zhang, Liankuan Tao, Haiyang Yu, Xiang Chen, Xin Xu, Xi Tian, Lei Li, Zhoubo Li, Shumin Deng, Yunzhi Yao, Hongbin Ye, Xin Xie, Guozhou Zheng, Huajun Chen Zhejiang University: Ningyu Zhang, Liankuan Tao, Haiyang Yu, Xiang Chen, Xin Xu, Xi Tian, Lei Li, Zhoubo Li, Shumin Deng, Yunzhi Yao, Hongbin Ye, Xin Xie, Guozhou Zheng, Huajun Chen

100
README_CN.md
View File

@ -47,9 +47,9 @@ DeepKE包括了三个模块，可以进行命名实体识别、关系抽取以
DeepKE支持pip安装使用，以常规全监督设定关系抽取为例，经过以下五个步骤就可以实现一个常规关系抽取模型 DeepKE支持pip安装使用，以常规全监督设定关系抽取为例，经过以下五个步骤就可以实现一个常规关系抽取模型
**Step 1** 下载代码 ```git clone https://github.com/zjunlp/DeepKE.git```（别忘记star和fork哈！！！） **Step 1**：下载代码 ```git clone https://github.com/zjunlp/DeepKE.git```（别忘记star和fork哈！！！）
**Step 2** 使用anaconda创建虚拟环境，进入虚拟环境(提供Dockerfile源码可自行创建镜像，位于docker文件夹中) **Step 2**：使用anaconda创建虚拟环境，进入虚拟环境(提供Dockerfile源码可自行创建镜像，位于docker文件夹中)
``` ```
conda create -n deepke python=3.8 conda create -n deepke python=3.8
@ -70,24 +70,26 @@ python setup.py install
python setup.py develop python setup.py develop
``` ```
**Step 3** 进入任务文件夹，以常规关系抽取为例 **Step 3** ：进入任务文件夹，以常规关系抽取为例
``` ```
cd DeepKE/example/re/standard cd DeepKE/example/re/standard
``` ```
**Step 4** 模型训练，训练用到的参数可在conf文件夹内修改 **Step 4** ：模型训练，训练用到的参数可在conf文件夹内修改
``` ```
python run.py python run.py
``` ```
**Step 5** 模型预测。预测用到的参数可在conf文件夹内修改 **Step 5** ：模型预测。预测用到的参数可在conf文件夹内修改
``` ```
python predict.py python predict.py
``` ```
<br>
### 环境依赖 ### 环境依赖
> python == 3.8 > python == 3.8
@ -118,11 +120,19 @@ python predict.py
| 秦始皇兵马俑位于陕西省西安市，1961年被国务院公布为第一批全国重点文物保护单位，是世界八大奇迹之一。 | 秦始皇 | 陕西省，西安市 | 国务院 | | 秦始皇兵马俑位于陕西省西安市，1961年被国务院公布为第一批全国重点文物保护单位，是世界八大奇迹之一。 | 秦始皇 | 陕西省，西安市 | 国务院 |
- 具体流程请进入详细的README中 - 具体流程请进入详细的README中
- **[常规全监督STANDARD](https://github.com/zjunlp/deepke/blob/main/example/ner/standard)** - **[常规全监督STANDARD](https://github.com/zjunlp/DeepKE/tree/main/example/ner/standard)**
**Step1**: 进入`DeepKE/example/ner/standard`，数据集和参数配置可以分别在`data`和`conf`文件夹中修改；<br> **Step1**: 进入`DeepKE/example/ner/standard`，下载数据集
**Step2**: 模型训练 ```bash
wget 120.27.214.45/Data/ner/standard/data.tar.gz
tar -xzvf data.tar.gz
```
**Step2**: 模型训练<br>
数据集和参数配置可以分别在`data`和`conf`文件夹中修改
``` ```
python run.py python run.py
@ -135,9 +145,17 @@ python predict.py
- **[少样本FEW-SHOT](https://github.com/zjunlp/DeepKE/tree/main/example/ner/few-shot)** - **[少样本FEW-SHOT](https://github.com/zjunlp/DeepKE/tree/main/example/ner/few-shot)**
**Step1**: 进入`DeepKE/example/ner/few-shot`，模型加载和保存位置以及参数配置可以在`conf`文件夹中修改；<br> **Step1**: 进入`DeepKE/example/ner/few-shot`，下载数据集
**Step2**：模型训练，默认使用`CoNLL-2003`数据集进行训练 ```bash
wget 120.27.214.45/Data/ner/few_shot/data.tar.gz
tar -xzvf data.tar.gz
```
**Step2**：低资源场景下训练模型<br>
模型加载和保存位置以及参数配置可以在`conf`文件夹中修改
``` ```
python run.py +train=few_shot python run.py +train=few_shot
@ -162,11 +180,19 @@ python predict.py
| 提起杭州的美景，西湖总是第一个映入脑海的词语。 | 所在城市 | 西湖 | 8 | 杭州 | 2 | | 提起杭州的美景，西湖总是第一个映入脑海的词语。 | 所在城市 | 西湖 | 8 | 杭州 | 2 |
- 具体流程请进入详细的README中，RE包括了以下三个子功能 - 具体流程请进入详细的README中，RE包括了以下三个子功能
- **[常规全监督STANDARD](https://github.com/zjunlp/deepke/blob/main/example/re/standard)** - **[常规全监督STANDARD](https://github.com/zjunlp/DeepKE/tree/main/example/re/standard)**
**Step1**：进入`DeepKE/example/re/standard`，数据集和参数配置可以分别进入`data`和`conf`文件夹中修改；<br> **Step1**：进入`DeepKE/example/re/standard`，下载数据集
**Step2**：模型训练 ```bash
wget 120.27.214.45/Data/re/standard/data.tar.gz
tar -xzvf data.tar.gz
```
**Step2**：模型训练<br>
数据集和参数配置可以分别进入`data`和`conf`文件夹中修改
``` ```
python run.py python run.py
@ -178,11 +204,21 @@ python predict.py
python predict.py python predict.py
``` ```
- **[少样本FEW-SHOT](https://github.com/zjunlp/deepke/blob/main/example/re/few-shot)** - **[少样本FEW-SHOT](https://github.com/zjunlp/DeepKE/tree/main/example/re/few-shot)**
**Step1**：进入`DeepKE/example/re/few-shot`，数据集和参数配置可以分别进入`data`和`conf`文件夹中修改；<br> **Step1**：进入`DeepKE/example/re/few-shot`，下载数据集
**Step2**：模型训练，如需从上次训练的模型开始训练：设置`conf/train.yaml`中的`train_from_saved_model`为上次保存模型的路径，每次训练的日志默认保存在根目录，可用`log_dir`来配置；<br> ```bash
wget 120.27.214.45/Data/re/few_shot/data.tar.gz
tar -xzvf data.tar.gz
```
**Step2**：模型训练<br>
- 数据集和参数配置可以分别进入`data`和`conf`文件夹中修改
- 如需从上次训练的模型开始训练：设置`conf/train.yaml`中的`train_from_saved_model`为上次保存模型的路径，每次训练的日志默认保存在根目录，可用`log_dir`来配置
``` ```
python run.py python run.py
@ -194,12 +230,20 @@ python predict.py
python predict.py python predict.py
``` ```
- **[文档级DOCUMENT](https://github.com/zjunlp/deepke/blob/main/example/re/document)** <br> - **[文档级DOCUMENT](https://github.com/zjunlp/DeepKE/tree/main/example/re/document)** <br>
```train_distant.json```由于文件太大，请自行从Google Drive上下载到data/目录下；<br>
**Step1**：进入`DeepKE/example/re/document`，数据集和参数配置可以分别进入`data`和`conf`文件夹中修改；<br> **Step1**：进入`DeepKE/example/re/document`，下载数据集
**Step2**：模型训练，如需从上次训练的模型开始训练：设置`conf/train.yaml`中的`train_from_saved_model`为上次保存模型的路径，每次训练的日志默认保存在根目录，可用`log_dir`来配置； ```bash
wget 120.27.214.45/Data/re/document/data.tar.gz
tar -xzvf data.tar.gz
```
**Step2**：模型训练<br>
- 数据集和参数配置可以分别进入`data`和`conf`文件夹中修改
- 如需从上次训练的模型开始训练：设置`conf/train.yaml`中的`train_from_saved_model`为上次保存模型的路径，每次训练的日志默认保存在根目录，可用`log_dir`来配置；
``` ```
python run.py python run.py
@ -221,11 +265,19 @@ python predict.py
| 2014年10月1日许鞍华执导的电影《黄金时代》上映 | 上映时间 | 黄金时代 | 19 | 2014年10月1日 | 0 | | 2014年10月1日许鞍华执导的电影《黄金时代》上映 | 上映时间 | 黄金时代 | 19 | 2014年10月1日 | 0 |
- 具体流程请进入详细的README中 - 具体流程请进入详细的README中
- **[常规全监督STANDARD](https://github.com/zjunlp/deepke/blob/main/example/ae/standard)** - **[常规全监督STANDARD](https://github.com/zjunlp/DeepKE/tree/main/example/ae/standard)**
**Step1**：进入`DeepKE/example/re/standard`，数据集和参数配置可以分别进入`data`和`conf`文件夹中修改；<br> **Step1**：进入`DeepKE/example/re/standard`，下载数据集
**Step2**：模型训练 ```bash
wget 120.27.214.45/Data/ae/standard/data.tar.gz
tar -xzvf data.tar.gz
```
**Step2**：模型训练<br>
数据集和参数配置可以分别进入`data`和`conf`文件夹中修改
``` ```
python run.py python run.py
@ -237,6 +289,8 @@ python predict.py
python predict.py python predict.py
``` ```
<br>
### Notebook教程 ### Notebook教程
本工具提供了若干Notebook和Google Colab教程，用户可针对性调试学习。 本工具提供了若干Notebook和Google Colab教程，用户可针对性调试学习。