This commit is contained in:
tlk-dsg 2021-09-23 16:23:03 +08:00
parent 9c4ae6cbe5
commit 192fd6ccd8
4 changed files with 569 additions and 3 deletions
src/deepke/attribution_extraction/standard/tools
tutorial-notebooks/ae/standard

View File

@ -96,7 +96,7 @@ def preprocess(cfg):
test_data = load_csv(test_fp)
attribute_data = load_csv(attribute_fp)
logger.info('convert relation into index...')
logger.info('convert attribution into index...')
atts = _handle_attribute_data(attribute_data)
_add_attribute_data(atts,train_data)
_add_attribute_data(atts,test_data)

View File

@ -0,0 +1,565 @@
{
"cells": [
{
"cell_type": "markdown",
"source": [
"## attribution extraction experiment\n",
"> Tutorial author: 陶联宽(22051063@zju.edu.cn)\n",
"\n",
"On this demo, we use `pretrain_language model` to extract attributions.\n",
"We hope this demo can help you understand the process of construction knowledge graph and the principles and common methods of triplet extraction.\n",
"\n",
"This demo uses `Python3`.\n",
"\n",
"### Dataset\n",
"In this example,we get some Chinese text to extract the triples\n",
"\n",
"sentence|attribute|entity|entity_offset|attribute_value|attribute_value_offset\n",
":---:|:---:|:---:|:---:|:---:|:---:\n",
"苏轼10371101年字子瞻又字和仲号“东坡居士”眉州眉山即今四川眉州是宋代北宋著名的文学家、书画家|字|苏轼|0|和仲|21\n",
"阳成俊,男,汉族,贵州省委党校大学学历|民族|阳成俊|0|汉族|6\n",
"司马懿,字仲达,河南温县人|字|司马懿|0|仲达|6\n",
"\n",
"- train.csv: It contains 6 training triples,each lines represent one triple,sorted by sentence,attribute,entity,entity's offset,attribute value attribute value's offset,and separated by ,.\n",
"- valid.csv: It contains 2 training triples,each lines represent one triple,sorted by sentence,attribute,entity,entity's offset,attribute value attribute value's offset,and separated by ,.\n",
"- test.csv: It contains 2 training triples,each lines represent one triple,sorted by sentence,attribute,entity,entity's offset,attribute value attribute value's offset,and separated by ,.\n",
"- attribute.csv: It contains 3 attribute triples,each lines sorted by attribute,index and separated by ,."
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"### BERT \n",
"\n",
"![BERT](img/Bert.png)\n",
"\n",
"After Bert coding, the original sentence can get rich semantic information. The obtained results are input into the bidirectional LSTM, and the output results can obtain the relationship information of the sentence.\n"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"### Code experience\n",
"\n",
"Important tips\n",
"- When we use pretrain language model, we need to load about 500MB of model data,so it is more recommended to download to the local and run.At this time, you only need to add `lm_file` value to the address of the local folder. See the link [transformers](https://huggingface.co/transformers/)"
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# Run the neural network with pytorch and confirm whether it is installed before running\n",
"!pip install torch\n",
"!pip install matplotlib\n",
"!pip install transformers"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# import the whole modules\n",
"import os\n",
"import csv\n",
"import math\n",
"import pickle\n",
"import logging\n",
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from torch import optim\n",
"from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence\n",
"from torch.utils.data import Dataset,DataLoader\n",
"from sklearn.metrics import precision_recall_fscore_support\n",
"from typing import List, Tuple, Dict, Any, Sequence, Optional, Union\n",
"from transformers import BertTokenizer, BertModel\n",
"\n",
"logger = logging.getLogger(__name__)"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# Configuration file of model parameters\n",
"class Config(object):\n",
" model_name = 'lm' # ['cnn', 'gcn', 'lm', 'rnn']\n",
" use_pcnn = True\n",
" min_freq = 1\n",
" pos_limit = 20\n",
" out_path = 'data/out' \n",
" batch_size = 2 \n",
" word_dim = 10\n",
" pos_dim = 5\n",
" dim_strategy = 'sum' # ['sum', 'cat']\n",
" out_channels = 20\n",
" intermediate = 10\n",
" kernel_sizes = [3, 5, 7]\n",
" activation = 'gelu'\n",
" pooling_strategy = 'max'\n",
" dropout = 0.3\n",
" epoch = 10\n",
" num_relations = 4\n",
" learning_rate = 3e-4\n",
" lr_factor = 0.7 # 学习率的衰减率\n",
" lr_patience = 3 # 学习率衰减的等待epoch\n",
" weight_decay = 1e-3 # L2正则\n",
" early_stopping_patience = 6\n",
" train_log = True\n",
" log_interval = 1\n",
" show_plot = True\n",
" only_comparison_plot = False\n",
" plot_utils = 'matplot'\n",
" lm_file = 'bert-base-chinese'\n",
" # lm_file = '/Users/yuhaiyang/transformers/bert-base-chinese'\n",
" lm_num_hidden_layers = 2\n",
" rnn_layers = 2\n",
" \n",
"cfg = Config()"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# Functions required for preprocessing\n",
"Path = str\n",
"\n",
"def load_csv(fp: Path, is_tsv: bool = False, verbose: bool = True) -> List:\n",
" if verbose:\n",
" logger.info(f'load csv from {fp}')\n",
"\n",
" dialect = 'excel-tab' if is_tsv else 'excel'\n",
" with open(fp, encoding='utf-8') as f:\n",
" reader = csv.DictReader(f, dialect=dialect)\n",
" return list(reader)\n",
"\n",
" \n",
"def load_pkl(fp: Path, verbose: bool = True) -> Any:\n",
" if verbose:\n",
" logger.info(f'load data from {fp}')\n",
"\n",
" with open(fp, 'rb') as f:\n",
" data = pickle.load(f)\n",
" return data\n",
"\n",
"\n",
"def save_pkl(data: Any, fp: Path, verbose: bool = True) -> None:\n",
" if verbose:\n",
" logger.info(f'save data in {fp}')\n",
"\n",
" with open(fp, 'wb') as f:\n",
" pickle.dump(data, f)\n",
" \n",
" \n",
"def _handle_attribute_data(attribute_data: List[Dict]) -> Dict:\n",
" atts = OrderedDict()\n",
" attribute_data = sorted(attribute_data, key=lambda i: int(i['index']))\n",
" for d in attribute_data:\n",
" atts[d['attribute']] = {\n",
" 'index': int(d['index'])\n",
" }\n",
" return atts\n",
"\n",
"\n",
"def _add_attribute_data(atts: Dict, data: List) -> None:\n",
" for d in data:\n",
" d['att2idx'] = atts[d['attribute']]['index']\n",
"\n",
"\n",
"def seq_len_to_mask(seq_len: Union[List, np.ndarray, torch.Tensor], max_len=None, mask_pos_to_true=True):\n",
" \n",
" if isinstance(seq_len, list):\n",
" seq_len = np.array(seq_len)\n",
"\n",
" if isinstance(seq_len, np.ndarray):\n",
" seq_len = torch.from_numpy(seq_len)\n",
"\n",
" if isinstance(seq_len, torch.Tensor):\n",
" assert seq_len.dim() == 1, logger.error(f\"seq_len can only have one dimension, got {seq_len.dim()} != 1.\")\n",
" batch_size = seq_len.size(0)\n",
" max_len = int(max_len) if max_len else seq_len.max().long()\n",
" broad_cast_seq_len = torch.arange(max_len).expand(batch_size, -1).to(seq_len.device)\n",
" if mask_pos_to_true:\n",
" mask = broad_cast_seq_len.ge(seq_len.unsqueeze(1))\n",
" else:\n",
" mask = broad_cast_seq_len.lt(seq_len.unsqueeze(1))\n",
" else:\n",
" raise logger.error(\"Only support 1-d list or 1-d numpy.ndarray or 1-d torch.Tensor.\")\n",
"\n",
" return mask\n",
"\n",
"\n",
"def _lm_serialize(data: List[Dict], cfg):\n",
" logger.info('use bert tokenizer...')\n",
" tokenizer = BertTokenizer.from_pretrained(cfg.lm_file)\n",
" for d in data:\n",
" sent = d['sentence'].strip()\n",
" sent += '[SEP]' + d['entity'] + '[SEP]' + d['attribute_value']\n",
" d['token2idx'] = tokenizer.encode(sent, add_special_tokens=True)\n",
" d['seq_len'] = len(d['token2idx'])"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# Preprocess\n",
"logger.info('load raw files...')\n",
"train_fp = os.path.join('data/train.csv')\n",
"valid_fp = os.path.join('data/valid.csv')\n",
"test_fp = os.path.join('data/test.csv')\n",
"attribute_fp = os.path.join('data/attribute.csv')\n",
"\n",
"train_data = load_csv(train_fp)\n",
"valid_data = load_csv(valid_fp)\n",
"test_data = load_csv(test_fp)\n",
"attribute_data = load_csv(attribute_fp)\n",
"\n",
"for d in train_data:\n",
" d['tokens'] = eval(d['tokens'])\n",
"for d in valid_data:\n",
" d['tokens'] = eval(d['tokens'])\n",
"for d in test_data:\n",
" d['tokens'] = eval(d['tokens'])\n",
" \n",
"llogger.info('convert attribution into index...')\n",
"atts = _handle_attribute_data(attribute_data)\n",
"_add_attribute_data(atts,train_data)\n",
"_add_attribute_data(atts,test_data)\n",
"_add_attribute_data(atts,valid_data)\n",
"\n",
"logger.info('verify whether use pretrained language models...')\n",
"\n",
"logger.info('use pretrained language models serialize sentence...')\n",
"_lm_serialize(train_data, cfg)\n",
"_lm_serialize(valid_data, cfg)\n",
"_lm_serialize(test_data, cfg)\n",
"\n",
"logger.info('save data for backup...')\n",
"os.makedirs(cfg.out_path, exist_ok=True)\n",
"train_save_fp = os.path.join(cfg.out_path, 'train.pkl')\n",
"valid_save_fp = os.path.join(cfg.out_path, 'valid.pkl')\n",
"test_save_fp = os.path.join(cfg.out_path, 'test.pkl')\n",
"save_pkl(train_data, train_save_fp)\n",
"save_pkl(valid_data, valid_save_fp)\n",
"save_pkl(test_data, test_save_fp)"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# pytorch construct Dataset\n",
"def collate_fn(cfg):\n",
" def collate_fn_intra(batch):\n",
" batch.sort(key=lambda data: data['seq_len'], reverse=True)\n",
" max_len = batch[0]['seq_len']\n",
"\n",
" def _padding(x, max_len):\n",
" return x + [0] * (max_len - len(x))\n",
"\n",
" x, y = dict(), []\n",
" word, word_len = [], []\n",
" head_pos, tail_pos = [], []\n",
" pcnn_mask = []\n",
" for data in batch:\n",
" word.append(_padding(data['token2idx'], max_len))\n",
" word_len.append(data['seq_len'])\n",
" y.append(int(data['att2idx']))\n",
"\n",
" if cfg.model_name != 'lm':\n",
" head_pos.append(_padding(data['entity_pos'], max_len))\n",
" tail_pos.append(_padding(data['attribute_value_pos'], max_len))\n",
" if cfg.model_name == 'cnn':\n",
" if cfg.use_pcnn:\n",
" pcnn_mask.append(_padding(data['entities_pos'], max_len))\n",
"\n",
" x['word'] = torch.tensor(word)\n",
" x['lens'] = torch.tensor(word_len)\n",
" y = torch.tensor(y)\n",
"\n",
" if cfg.model_name != 'lm':\n",
" x['entity_pos'] = torch.tensor(head_pos)\n",
" x['attribute_value_pos'] = torch.tensor(tail_pos)\n",
" if cfg.model_name == 'cnn' and cfg.use_pcnn:\n",
" x['pcnn_mask'] = torch.tensor(pcnn_mask)\n",
" if cfg.model_name == 'gcn':\n",
" # 没找到合适的做 parsing tree 的工具,暂时随机初始化\n",
" B, L = len(batch), max_len\n",
" adj = torch.empty(B, L, L).random_(2)\n",
" x['adj'] = adj\n",
" return x, y\n",
"\n",
" return collate_fn_intra\n",
"\n",
"\n",
"class CustomDataset(Dataset):\n",
" \"\"\"\n",
" 默认使用 List 存储数据\n",
" \"\"\"\n",
" def __init__(self, fp):\n",
" self.file = load_pkl(fp)\n",
"\n",
" def __getitem__(self, item):\n",
" sample = self.file[item]\n",
" return sample\n",
"\n",
" def __len__(self):\n",
" return len(self.file)"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# pretrain language model\n",
"class PretrainLM(nn.Module):\n",
" def __init__(self, cfg):\n",
" super(PretrainLM, self).__init__()\n",
" self.num_layers = cfg.rnn_layers\n",
" self.lm = BertModel.from_pretrained(cfg.lm_file, num_hidden_layers=cfg.lm_num_hidden_layers)\n",
" self.bilstm = nn.LSTM(768,10,batch_first=True,bidirectional=True,num_layers=cfg.rnn_layers,dropout=cfg.dropout)\n",
" self.fc = nn.Linear(20, cfg.num_relations)\n",
"\n",
" def forward(self, x):\n",
" N = self.num_layers\n",
" word, lens = x['word'], x['lens']\n",
" B = word.size(0)\n",
" output, pooler_output = self.lm(word)\n",
" output = pack_padded_sequence(output, lens, batch_first=True, enforce_sorted=True)\n",
" _, (output,_) = self.bilstm(output)\n",
" output = output.view(N, 2, B, 10).transpose(1, 2).contiguous().view(N, B, 20).transpose(0, 1)\n",
" output = output[:,-1,:]\n",
" output = self.fc(output)\n",
" \n",
" return output"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# p,r,f1 measurement\n",
"class PRMetric():\n",
" def __init__(self):\n",
" \n",
" self.y_true = np.empty(0)\n",
" self.y_pred = np.empty(0)\n",
"\n",
" def reset(self):\n",
" self.y_true = np.empty(0)\n",
" self.y_pred = np.empty(0)\n",
"\n",
" def update(self, y_true:torch.Tensor, y_pred:torch.Tensor):\n",
" y_true = y_true.cpu().detach().numpy()\n",
" y_pred = y_pred.cpu().detach().numpy()\n",
" y_pred = np.argmax(y_pred,axis=-1)\n",
"\n",
" self.y_true = np.append(self.y_true, y_true)\n",
" self.y_pred = np.append(self.y_pred, y_pred)\n",
"\n",
" def compute(self):\n",
" p, r, f1, _ = precision_recall_fscore_support(self.y_true,self.y_pred,average='macro',warn_for=tuple())\n",
" _, _, acc, _ = precision_recall_fscore_support(self.y_true,self.y_pred,average='micro',warn_for=tuple())\n",
"\n",
" return acc,p,r,f1"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# Iteration in training process\n",
"def train(epoch, model, dataloader, optimizer, criterion, cfg):\n",
" model.train()\n",
"\n",
" metric = PRMetric()\n",
" losses = []\n",
"\n",
" for batch_idx, (x, y) in enumerate(dataloader, 1):\n",
" optimizer.zero_grad()\n",
" y_pred = model(x)\n",
" loss = criterion(y_pred, y)\n",
"\n",
" loss.backward()\n",
" optimizer.step()\n",
"\n",
" metric.update(y_true=y, y_pred=y_pred)\n",
" losses.append(loss.item())\n",
"\n",
" data_total = len(dataloader.dataset)\n",
" data_cal = data_total if batch_idx == len(dataloader) else batch_idx * len(y)\n",
" if (cfg.train_log and batch_idx % cfg.log_interval == 0) or batch_idx == len(dataloader):\n",
" acc,p,r,f1 = metric.compute()\n",
" print(f'Train Epoch {epoch}: [{data_cal}/{data_total} ({100. * data_cal / data_total:.0f}%)]\\t'\n",
" f'Loss: {loss.item():.6f}')\n",
" print(f'Train Epoch {epoch}: Acc: {100. * acc:.2f}%\\t'\n",
" f'macro metrics: [p: {p:.4f}, r:{r:.4f}, f1:{f1:.4f}]')\n",
"\n",
" if cfg.show_plot and not cfg.only_comparison_plot:\n",
" if cfg.plot_utils == 'matplot':\n",
" plt.plot(losses)\n",
" plt.title(f'epoch {epoch} train loss')\n",
" plt.show()\n",
"\n",
" return losses[-1]\n",
"\n",
"\n",
"# Iteration in testing process\n",
"def validate(epoch, model, dataloader, criterion,verbose=True):\n",
" model.eval()\n",
"\n",
" metric = PRMetric()\n",
" losses = []\n",
"\n",
" for batch_idx, (x, y) in enumerate(dataloader, 1):\n",
" with torch.no_grad():\n",
" y_pred = model(x)\n",
" loss = criterion(y_pred, y)\n",
"\n",
" metric.update(y_true=y, y_pred=y_pred)\n",
" losses.append(loss.item())\n",
"\n",
" loss = sum(losses) / len(losses)\n",
" acc,p,r,f1 = metric.compute()\n",
" data_total = len(dataloader.dataset)\n",
" if verbose:\n",
" print(f'Valid Epoch {epoch}: [{data_total}/{data_total}](100%)\\t Loss: {loss:.6f}')\n",
" print(f'Valid Epoch {epoch}: Acc: {100. * acc:.2f}%\\tmacro metrics: [p: {p:.4f}, r:{r:.4f}, f1:{f1:.4f}]\\n\\n')\n",
"\n",
" return f1,loss"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# Load dataset\n",
"train_dataset = CustomDataset(train_save_fp)\n",
"valid_dataset = CustomDataset(valid_save_fp)\n",
"test_dataset = CustomDataset(test_save_fp)\n",
"\n",
"train_dataloader = DataLoader(train_dataset, batch_size=cfg.batch_size, shuffle=True, collate_fn=collate_fn(cfg))\n",
"valid_dataloader = DataLoader(valid_dataset, batch_size=cfg.batch_size, shuffle=True, collate_fn=collate_fn(cfg))\n",
"test_dataloader = DataLoader(test_dataset, batch_size=cfg.batch_size, shuffle=True, collate_fn=collate_fn(cfg))"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# main entry, define optimization function, loss function and so on\n",
"# start epoch\n",
"# Use the loss of the valid dataset to make an early stop judgment. When it does not decline, this is the time when the model generalization is the best.\n",
"model = PretrainLM(cfg)\n",
"print(model)\n",
"\n",
"optimizer = optim.Adam(model.parameters(), lr=cfg.learning_rate, weight_decay=cfg.weight_decay)\n",
"scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, factor=cfg.lr_factor, patience=cfg.lr_patience)\n",
"criterion = nn.CrossEntropyLoss()\n",
"\n",
"best_f1, best_epoch = -1, 0\n",
"es_loss, es_f1, es_epoch, es_patience, best_es_epoch, best_es_f1, = 1000, -1, 0, 0, 0, -1\n",
"train_losses, valid_losses = [], []\n",
"\n",
"logger.info('=' * 10 + ' Start training ' + '=' * 10)\n",
"for epoch in range(1, cfg.epoch + 1):\n",
" train_loss = train(epoch, model, train_dataloader, optimizer, criterion, cfg)\n",
" valid_f1, valid_loss = validate(epoch, model, valid_dataloader, criterion)\n",
" scheduler.step(valid_loss)\n",
"\n",
" train_losses.append(train_loss)\n",
" valid_losses.append(valid_loss)\n",
" if best_f1 < valid_f1:\n",
" best_f1 = valid_f1\n",
" best_epoch = epoch\n",
" if es_loss > valid_loss:\n",
" es_loss = valid_loss\n",
" es_f1 = valid_f1\n",
" best_es_f1 = valid_f1\n",
" es_epoch = epoch\n",
" best_es_epoch = epoch\n",
" es_patience = 0\n",
" else:\n",
" es_patience += 1\n",
" if es_patience >= cfg.early_stopping_patience:\n",
" best_es_epoch = es_epoch\n",
" best_es_f1 = es_f1\n",
"\n",
"if cfg.show_plot:\n",
" if cfg.plot_utils == 'matplot':\n",
" plt.plot(train_losses, 'x-')\n",
" plt.plot(valid_losses, '+-')\n",
" plt.legend(['train', 'valid'])\n",
" plt.title('train/valid comparison loss')\n",
" plt.show()\n",
"\n",
"\n",
"print(f'best(valid loss quota) early stopping epoch: {best_es_epoch}, '\n",
" f'this epoch macro f1: {best_es_f1:0.4f}')\n",
"print(f'total {cfg.epoch} epochs, best(valid macro f1) epoch: {best_epoch}, '\n",
" f'this epoch macro f1: {best_f1:.4f}')\n",
"\n",
"test_f1, _ = validate(0, model, test_dataloader, criterion,verbose=False)\n",
"print(f'after {cfg.epoch} epochs, final test data macro f1: {test_f1:.4f}')"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"This demo does not include parameter adjustment. Interested students can go to [deepke] by themselves http://openkg.cn/tool/deepke Warehouse, download and use more models:)"
],
"metadata": {}
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

View File

@ -6,7 +6,8 @@
"## attribution extraction experiment\n",
"> Tutorial author: 陶联宽(22051063@zju.edu.cn)\n",
"\n",
"On this demo, we use `lstm` to extract attributions.We hope this demo can help you understand the process of construction knowledge graph and the principles and common methods of triplet extraction.\n",
"On this demo, we use `lstm` to extract attributions.\n",
"We hope this demo can help you understand the process of construction knowledge graph and the principles and common methods of triplet extraction.\n",
"\n",
"This demo uses `Python3`.\n",
"\n",
@ -302,7 +303,7 @@
"for d in test_data:\n",
" d['tokens'] = eval(d['tokens'])\n",
"\n",
"logger.info('convert relation into index...')\n",
"logger.info('convert attribution into index...')\n",
"atts = _handle_attribute_data(attribute_data)\n",
"_add_attribute_data(atts,train_data)\n",
"_add_attribute_data(atts,test_data)\n",

Binary file not shown.

After

(image error) Size: 68 KiB