Apache License 2.0

技术解读

以下内容由 GitRead 中文社区 编辑团队基于项目 README 与官方资料整理, 结合中文开发者的常见使用场景补充技术选型建议。具体以项目官方文档为准。

decal/werdlists 深度技术分析

项目背景与痛点

在软件安全测试领域,拥有丰富、全面的数据集是编写有效的测试用例的基础。然而,高质量的数据集并不容易获取。decal/werdlists 项目提供了多种词典、字典和其他数据集,旨在填补这一空白。这类数据集不仅有助于实现覆盖率更高的测试,还可以提升安全测试的质量,提高发现潜在漏洞的能力。在实际应用中,开发者可以利用这些数据集快速生成测试用例,并针对不同的应用场景做出灵活调整。

核心技术揭秘

decal/werdlists 项目采用的是模块化的数据管理和分发机制,这样的设计使得项目可以轻松维护和扩展。其核心架构包括数据集的生成、存储、分类和检索等模块。数据集经过分类整理后,用户可以根据不同需求快速找到合适的资源。

数据处理流程可抽象为以下部分:

  1. 数据获取:覆盖多种来源的数据采集工具和脚本,确保数据源的多样性和可靠性。
  2. 预处理:清洗和转换原始数据,使得数据适合用于测试用例生成。
  3. 存储与分类:将预处理后的数据进行分类存储,便于后续使用。
  4. 数据检索:提供方便快捷的数据检索接口,确保用户可以根据需求高效查找和使用。

以下是一个简化版的数据流图,展示了数据如何从采集到最终被用户使用的整个过程。

graph TD A[数据来源] --> B[数据获取与清洗] B --> C[数据预处理] C --> D[数据分类与存储] D --> E[数据检索] E --> F[用户使用]

功能亮点与差异

相较于同类数据集项目,decal/werdlists 的独特之处在于其庞大的数据覆盖率以及灵活多变的数据处理能力。在数据集的生成和维护方面,项目不仅采用了许多常见的技术和资源,还引入了新的数据采集技术和算法,以增强数据的可靠性和新颖性。另外,项目的开源性质确保了社区的广泛参与和共同演化。此外,良好的文档和支持使得新手也能迅速上手,降低了使用复杂性。

应用场景与落地建议

在生产环境使用此类数据集时,须注意以下几点:

  • 确保数据的时效性和动态更新能力。安全威胁的持续变化要求数据集也需要不断更新。
  • 考虑数据集的使用权限和许可问题,确保使用合规。
  • 监控数据检索性能,保证产品效率。

这种数据集适合用于以下场景:

  • 代码审计和漏洞扫描。
  • 编写自动化安全测试。
  • 网络攻防对抗演练。

综合评价

优点:

  • 高度模块化的架构设计带来了强大的扩展性和可维护性。
  • 数据种类丰富,可以广泛满足各类安全测试需求。
  • 开放的社区环境促进持续创新。

缺点:

  • 数据源的可靠性和数据质量取决于数据采集策略,用户可能需要自行评估。
  • 学习成本相对较高,特别是对初学者来说,理解数据集背后的逻辑和使用方式需要一定的技术积累。

综上,decal/werdlists 是一个值得推荐的安全测试数据集项目。它不仅提供了丰富的数据资源,而且通过开放的社区协作增强了项目的可持续发展能力。

关于

GitHub 原始描述(英文)

:keyboard: Wordlists, Dictionaries and Other Data Sets for Writing Software Security Test Cases

描述取自 GitHub 官方 API,保持仓库作者书写的原文。

语言占比

AGS Script
C
C++
HTML
JavaScript
Python
Ruby
Shell

README

Twitter: @decalresponds Ask Me Anything! werdlists stars forks Apache License 2.0 repo-size last commit watchers made-with-bash PayPal


werdlists


“Word Lists” for Software Security Test Cases

Word lists, Dictionary Files, Attack Strings, Miscellaneous Datasets and Proof-of-Concept Test Cases With a Collection of Tools for Penetration Testers

Brief Introduction to werdlists :scissors:

This project is a collection of word lists–they are mostly whitespace-delimited or line-based. Although the passes-dicts folder contains inputs for password cracking, overall the files amassed here are intended to be useful in facilitating the creation of insecure program state (with the help of a black-box fuzzer or scanning tool.) The vast majority of files are simply ASCII with the UNIX style newline. Beware that this project does not attempt in any way to be minimalist or lack verbosity!

Inspiration Taken From Similar Projects :thought_balloon:

werdlists is very similar to fuzzdb and SecLists. SecLists is maintained by my former colleague at IOActive, Daniel Miessler. Admittedly, werdlists is quite similar in mission as it’s a centralized attack strings and input data resource. Regardless, werdlists expands on a number of concepts: it has its own unique style, organization, original hand-crafted contents, dataset creation/management/validation scripts, scanner springboards, etc.

Unique Features Only Available With werdlists :100:

werdlists cross-references between the code repositories of third-party scanners and its own datasets that each tool will benefit from. Moreover, there are specialized parsing scripts exclusive to werdlists that extract results produced through pairing test tools with its own data. Output strings are gathered from those results and fed back into the test tools. In other words, there are a number of interactive and/or tunable feedback loops implemented. Quite a few of the werdlists data files were created this way.

Repository Directory Hierarchy and Structure :nut_and_bolt:

The scripts folder consists of shell scripts used for repository maintenance. There is a sub-directory of scripts called init where scripts that initialize data files are stored. If a script filename stored in init contains two dashes, then it’s output should reflect the contents of the associated data file. For example, compare manpages-environ and clib-package-names. All scripts were written using bash syntax. The contrib folder is for storing scripts contributed via pull request and the utils folder contains utilities that aren’t necessarily specific to the werdlists project, such as scripts for managing any wordlist file. Other data files were manually composed by hand and a small handful were created by recycling output strings back into input parameter lists, i.e. dirbdirs-feedback The tools folder lists security tools that the datasets contained in this repository can be provided as input for. Individual folders are detailed in the Folder Names and Description of Contents section below. All files in each dataset directory are detailed in the local README.md file for that folder (as opposed to the global README.md in the root directory being read now.)

Naming Scheme, Syntax and Meaning :speech_balloon:

Most files have the *.txt extension signifying the text/plain MIME type Often used formats besides plain text include: Comma-Separated Values (text/csv), Extended Markup Language (application/xml), Hyper Text Markup Language (application/html), etc. Any file that is larger than 1MB uncompressed will be compressed with xz according to the commands in the scripts/xzlarge-files bash script. Other file extensions in use are: *.ans, *.asc, *.bin, *.c, *.conf, *.cpp, *.csv, *.html, *.inf, *.ini, *.json, *.md, *.rpz, *.rst, *.sh, *.txt, *.xml, *.yaml, *.yml, *.zip, and *.zone.

Folder Names and Description of Contents :clipboard:

    Folder  Name     Description of Contents
apple-paths :rocket: Pathnames found on MacOS file systems
apple-data :apple: Data identifiers and such from Apple’s MacOS operating system
arpa-headers :email: Header fields transmitted over RFC2822 style protocols like SMTP
ascii-art :art: “Low bit” a.k.a. 7-bit ASCII art items without control characters
biology-info :microscope: Reference information useful in the study of biological issues
browser-data :door: Data related to GUI browser software like Chrome, FireFox, etc.
cert-data :scroll: Information commonly utilized by cryptographic certificate materials
char-encodes :ideograph_advantage: Various character encodings provided by different locales/charsets
char-sequence :black_nib: various character sequences modeled after ctype.h
chat-data :open_mouth: Additional data on IRC, XMPP and other such messaging protocols
cipher-data :blowfish: Data denoting or used by cryptographic algorithm implementations
cmd-usage :hammer: Help text shown in a terminal when attempting to execute CLI programs
code-keywords :coffee: Computer language identifiers, reserved words and similar syntax
cpu-arch :factory: Low-level computer architecture and hardware subjects
crypt-output :sparkles: Cipher text string outputs created by cryptographic hash functions
database-strs :floppy_disk: Strings often encountered when working with database software
dns-domains :globe_with_meridians: A list of domains that may have been found in the live DNS tree at one point
dns-hostnames :flashlight: The host name part of an FQDN
dns-records :ticket: Data specific to RR’s in the DNS system
dns-servers :battery: Data provided to, produced by or related to DNS name servers
dns-toplevel :top: TLD’s or Top Level Domains in the uppermost part of the DNS hierarchy
environ-vars :tent: Environment variable names, settings, etc.
exploit-info :8ball: Technical information on exploitation of security vulnerabilities
file-extens :anchor: Stuff on Filename extensions, i.e. the part after the dot
file-specs :file_folder: File format specifications as distributed by vendor(s)/author(s)
ftp-data :outbox_tray: Various FTP datum from RFC’s and elsewhere
glibc-data :gear: Data taken from the source code of the GNU C Library
html-words :keyboard: Words not uncommon to come across when parsing HTML dialects
http-agents :flags: Software version banners for HTTP User Agents also known as browsers
http-headers :cookie: Header fields sent in requests/responses by browser/server software
http-methods :arrow_forward: Names Request methods browsers send in the first line of HTTP
http-params :abcd: Parameters browsers sometimes send when requesting server URI paths
http-security :closed_lock_with_key: HTTP security info such as Content Security Policy
http-servers :office: Information related to the usage of web server software
http-status :slot_machine: Numeric HTTP status codes in server reply as RFC7231 specifies
inet-addrs :electric_plug: Numeric Internet addresses a.k.a. IP addresses–mostly version 4
inet-routes :cloud: Data useful in the maintenance and use of an Internet routing table
inet-services :fountain: Lists of Internet protocols/daemons–similar to /etc/services
infosec-people :neckbeard: Noteworthy individuals known from information security communities
iso-codes :heavy_check_mark: Codes, numbers and such as standardized by ISO
java-data :sunny: Data found in or related to source code of programs written with Java
linux-data :keycap_ten: Data identifiers and such from the Linux operating system
linux-paths :paperclips: Pathnames found on file systems created by Linux installations
malware-iocs :skull: IOC for identification of malware infections
mobile-devs :iphone: Mobile device development for “handheld” form factors
net-attacks :hotsprings: Info about attacks on telecommunications and Internetworks
net-ifaces :desktop_computer: Detailed information which can be extracted from network interfaces
ntfs-paths :open_file_folder: File paths expected to be seen in NTFS folders
owasp-data :honeybee: Data from or for OWASP
passes-dicts :key: Dictionary files for brute-force attacks against account passwords
passes-sites :unlock: Hashed or unencrypted passwords that were publicized after the breach of a well-known site
perl-data :camel: Data often seen in PERL (Practical Extraction and Report Language)
php-data :page_facing_up: Files containing information about the PHP programming language
postal-data :mailbox_with_mail: United States Postal Service information
python-data :snake: Data used by the Python scripting language interpreter at runtime
radio-data :radio: Things commonly used in radio frequency transmissions
regex-data :speech_balloon: Regular expression patterns used to launch/detect attacks
ruby-data :gem: Data typically seen within the syntax of the Ruby scripting language
search-dorks :mag_right: General purpose search-engine queries likely to find insecure sites
smtp-messages :envelope: Messages (i.e. signatures, auto-replies, etc.) sent by SMTP servers
soap-messages :incoming_envelope: SOAP (Simple Object Access Protocol) messages
social-data :eyes: Sociological or social media related data sets including logins and user names
software-strs :minidisc: Strings describing software engineering, programming languages, etc.
string-enums :ferris_wheel: Enumerations of values that aren’t too terribly unusual
system-admin :necktie: System administration and BOFH related materials
system-notices :warning: Disclaimer/warning messages shown by networked computer systems
telco-data :telephone_receiver: Voice telecommunications technologies: POTS, PCS, VoIP, SMS etc.
text-files :pushpin: zine articles and such like those archived at Jason Scott’s textfiles.com
text-words :pencil2: Lists of words likely to be found in an actual hard copy dictionary
top-secret :alien: Files and/or data related to documents that were/are classified
unicode-data :symbols: Unicode character usage and representation
unix-data :shell: Data associated with various flavors of the UNIX OS and its clones
unix-paths :file_cabinet: File path names found in various UNIX file systems
uri-attacks :boom: Malicious URI materials specially crafted for attack targets
uri-schemes :paperclip: Lists containing references for URI schemes (part before colon)
uri-data :link: Universal Resource Identifier related data
vuln-data :bar_chart: Information about security vulnerabilities found in server software
webapp-attacks :syringe: Proof-of-concept samples demonstrating attacks against web applications
webapp-data :briefcase: Data associated with applications hosted on web servers
webapp-dirs :bookmark_tabs: Directories related to applications running on a web server
webapp-files :card_index: Files related to applications running on a web server
webapp-paths :surfer: Path names related to applications running on a web server
webapp-words :thought_balloon: Words related to applications running on a web server
web-sites :earth_americas: Addresses to and/or information on significant WWW sites
wifi-networks :satellite: IEEE 802.11 Wi-Fi network information
windows-data :briefcase: Data only found within the Microsoft Windows series of OSes

ans asc bin c conf cpp csv html inf ini json md rpz rst sh txt xml yaml yml zip zone

评论

暂无评论