AutoPatent

Note: We will make the data and code available upon the paper's acceptance.

AutoPatent: A Multi-Agent Framework for Automatic Patent Generation

Qiyao Wang^1,2, Shiwen Ni¹, Guhong Chen^1,3, Huaren Liu², Shule Lu²,
Xi Feng¹, Chi Wei¹, Qiang Qu¹, Hamid Alinejad-Rokny⁵, Chengming Li⁶, Yuan Lin^2†, Min Yang^1,4†
*Equal Contribution, † Corresponding Authors.
¹Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
²Dalian University of Technology
³Southern University of Science and Technology
⁴Shenzhen University of Advanced Technology
⁵The University of New South Wales
⁶Shenzhen MSU-BIT University

📢 News

• [2024-08-30] Research Beginning.
• [2024-12-13] We have submitted our paper to arXiv, and it will be publicly available soon.
• [2024-12-13] We have released the initial version of AutoPatent on GitHub. The complete code and data will be made publicly available following the paper’s acceptance.
• [2024-12-16] The paper has been public at Arxiv.
• [2024-12-20] This work has been fortunate to receive attention and coverage from Xin Zhi Yuan, and it will continue to be expanded and improved in the future. [News]
• [2024-12-20] We have released 10 generated patent samples in this GitHub repository. Please see the “example” folder.
• [2025-03-09] We have added the few-shot experimental results.

Table of Contents

• Overview
• Quick Start
• Experiment Detail
  • Metric
    • Objective Metric
    • Human Evaluation
  • Compared Method
    • Zero-Shot Prompting
    • Few-Shot Prompting
    • Supervised Fine-Tuning
  • Results
    • Objective Metric Results
    • Human Evaluation Results
• Demo
• Citation
• Contact

📖 Overview

We introduce a novel and practical task known as Draft2Patent, along with its corresponding D2P benchmark, which challenges LLMs to generate full-length patents averaging 17K tokens based on initial drafts. Patents present a significant challenge to LLMs due to their specialized nature, standardized terminology, and extensive length.

We propose a multi-agent framework called AutoPatent which leverages the LLM-based planner agent, writer agents, and examiner agent with PGTree and RRAG to generate to craft lengthy, intricate, and high-quality complete patent documents.

🧐 Quick Start

❗️ We will make the data and code available upon the paper’s acceptance.

Experiment Detail

Metric

Objective Metric

We use the n-gram-based metric, BLEU, the F1 scores of ROUGE-1, ROUGE-2, and ROUGE-L as the objective metrics.

We propose a new metric, termed IRR (Inverse Repetition Rate), to measure the degree of sentence repetition within the patent $\mathcal{P}=\{s_i|1\le i\le n\}$, which consists of $n$ sentences.

The IRR is defined as:

$$ IRR (\mathcal{P}, t) = \frac{C_n^2}{\sum_{i=1}^{n-1} \sum_{j=i+1}^{n} f(s_i, s_j) + \varepsilon} $$

Where the time complexity of the IRR metric is $O(n^2)$, $\varepsilon$ is a small value added for smoothing to prevent division by zero, and $t$ is threshold for determining whether two sentences, $s_i$ and $s_j$, are considered repetitions based on their Jaccard similarity $J$, calculated after removing stop words.

The function $f(s_i, s_j)$ is defined as:

$$ f(s_i, s_j) = \begin{cases} 1, & \text{if } J(s_i, s_j) \geq t, \ 0, & \text{if } J(s_i, s_j) < t. \end{cases} $$

Human Evaluation

We invite three experts who are familiar with the patent law and patent drafting to evaluate the quality of generated patent using a single-bind review.

Compared Method

Zero-Shot Prompting

The prompt is provided in Appendix C.1 of the paper.

Models:

• Commercial Model
  • GPT-4o
  • GPT-4o-mini
• Open source model
  • LLAMA3.1 (8B and 70B)
  • Qwen2.5 (7B, 14B, 32B and 72B)
  • Mistral-7B

Few-Shot Prompting

We leverage few-shot prompting as a stronger baseline, providing from one to three examples before target draft. The average input length exceeds 20K tokens for one-shot, 60K tokens for two-shot, and 80K tokens for three-shot. This challenge for models with smaller context windows, such as GPT-4o-mini with a maximum length of 16,384 tokens and Mistral with a maximum length of 32K tokens. In this situation, we opt to conduct this experiment on Qwen2.5-7B and LLAMA3.1-8B.

Supervised Fine-Tuning

We utilize 1,500 draft-patent pairs from D2P’s training set to perform fully supervised fine-tuning on LLAMA3.1-8B, Qwen2.5-7B, and Mistral-7B models (each with fewer than 14 billion parameters).

The fine-tuning process leverages LLaMA-Factory as the tool for efficiently fine-tuning models.

Results

Objective Metric Results

Human Evaluation Results

Demo

We have released the first demo video of AutoPatent on our website.

A publicly accessible and customizable demo will be available upon the paper’s acceptance.

Citation

If you find this repository helpful, please consider citing the following paper:

@article{wang2024autopatent,
  title={AutoPatent: A Multi-Agent Framework for Automatic Patent Generation},
  author={Wang, Qiyao and Ni, Shiwen and Liu, Huaren and Lu, Shule and Chen, Guhong and Feng, Xi and Wei, Chi and Qu, Qiang and Alinejad-Rokny, Hamid and Lin, Yuan and others},
  journal={arXiv preprint arXiv:2412.09796},
  year={2024}
}

Star History

Contact

If you have any questions, feel free to contact us at [email protected] or [email protected].