Trash

Trash

Introduction

Trash is a command-line toolkit for generating parsers, parsing input, and analyzing and transforming grammars and parse trees. It supports two workflows: generate interpreter tables and parse directly with Trash, or generate an ANTLR4 parser application for a supported target language.

dotnet trash interp compiles grammar parse trees into .interp files containing lexer and parser ATNs, together with .tokens vocabularies. These tables can be loaded by dotnet trash parse without generating or compiling a target-language parser. By default, interp bundles the tables on stdout for the next command; use -o to write them to a directory. Grammar front ends currently support:

• ABNF (.abnf) — a basic subset, compiled with character-level tokenization and RFC core rules.
• ANTLR4 (.g4) — ANTLR4 grammar syntax.
• ANTLR4 Experimental (G4X) (.g4x, .g4x) — experimental ANTLR4 grammar syntax for new features, including unconstrained rule names. Rule kind is determined by the grammar declaration rather than capitalization.
• Bison (.y) — a basic subset, paired with a separate ANTLR4 or G4X lexer grammar.
• Invisible XML (iXML) (.ixml) — a basic subset compiled with character-level tokenization to preserve scannerless matching.
• REx (.rex) — a basic subset of the EBNF notation used by the REx parser generator.
• W3C EBNF (.ebnf) — a basic subset, compiled with character-level tokenization to preserve scannerless matching.

For example, from the REx example directory:

dotnet trash parse Arithmetic.rex | dotnet trash interp |
  dotnet trash parse --allstar -i '1+2*3' | dotnet trash tree --text -a

Support for a notation does not imply support for all of its semantics. The ABNF, ANTLR4 Experimental (G4X), Bison, iXML, REx, and W3C EBNF examples document the supported subsets and current limitations. In particular, iXML interpretation produces Trash parse trees, not iXML’s prescribed XML output, and the interpreter does not execute arbitrary target-language actions or semantic predicates.

For generated applications, dotnet trash gen supplies ANTLR4 driver and build templates for supported targets. The grammars-v4 collection provides grammars for many programming languages and data formats.

Trash also provides XPath 4.0 and XQuery 4.0 expression processing, including XQuery Update operations, to query and edit parse trees. Preserved off-channel tokens, such as whitespace and comments, can participate in source-preserving transformations. Commands compose through shell pipelines using PAX/tar bundles of parse trees, diagnostics, and other artifacts; text-rendering commands can instead output source text, trees, XML, or Graphviz DOT.

Trash runs on Windows, Linux, and macOS as a .NET tool. Invoke its commands as dotnet trash <command>, for example dotnet trash parse --help. Interpreted parsing needs no separate target-language toolchain; generating and building parser applications requires the tools for the selected target.

Installation

Requirements

Install Dotnet 10.0.x

Install Globally

dotnet tool install -g trash

Uninstall

dotnet tool uninstall -g trash

Install Locally

dotnet new tool-manifest
dotnet tool install trash

List of commands

Every command is invoked as dotnet trash <name> [options] [args]. Most commands read a parse result set from stdin and write one to stdout, making them composable with |. Full invocation rules (argument order, -- separator, MSYS2 quoting) and a sortable reference table are in commands.html.

Command	Description
analyze	Analyze a grammar
caret	Caret operations on a parse tree
clonereplace	Clone and replace in a grammar
combine	Combine a split Antlr4 grammar
convert	Convert a grammar from one form to another
cover	Code coverage analysis
dot	Print a parse tree in Graphviz Dot format
extract	Extract target-specific code from a grammar
ff	Output FIRST and FOLLOW sets of a grammar
foldlit	Fold transform on grammar with literals
gen	Generate an Antlr4 parser for a given target language
genvsc	Generate VS Code extension files
glob	Expand glob file patterns
iconv	Convert file encoding
interp	Generate Antlr4 .interp files from a grammar parse tree
itext	Get strings from a PDF file
json	Print a parse tree in JSON structured format
nullable	Nullable analysis of a grammar
parse	Parse a grammar or use a generated parser to parse input
perf	Performance analysis of an Antlr grammar parse
query	Query parse trees using XPath
rename	Rename symbols in a grammar
sort	Sort rules in a grammar
split	Split a combined Antlr4 grammar
sponge	Write parse result set back to files on disk
text	Reconstruct source files into a PAX/tar bundle (--text for plain output)
tokens	Print tokens in a parse tree
tree	Render parse trees into a PAX/tar bundle (--text for plain output)
unfold	Unfold transform on a grammar
unfoldlit	Unfold transform with literals on a grammar
ungroup	Ungroup transform on a grammar
wdog	Kill a program that runs too long
xpath	Search parse trees using XPath 4.0
xml	Print a parse tree in XML format
xml2	Enumerate all XPath paths in a parse tree to leaves
xquery	Apply XQuery Update expressions to a parse tree

Examples

Runnable examples with step-by-step instructions are in the examples/ directory.

PAX/tar bundled data passing between commands

Trash commands pass an ordinary POSIX PAX/tar artifact bundle through standard input and standard output. Because it is a regular tar stream, a bundle can also be inspected with standard tools such as tar -tf and tar -xvf.

Each input parsed by dotnet trash parse normally contributes two regular-file members:

• <name>.pt contains one JSON-serialized ParsingResultSet object. It holds the parse-tree nodes, parser and lexer metadata, source name, and source text for that input.
• <name>.errors contains the parser diagnostics as UTF-8 text. It is empty when the input parsed without errors.

For example, parsing examples/a.g4 and examples/nested/b.g4 can produce:

a.g4.pt
a.g4.errors
nested/b.g4.pt
nested/b.g4.errors

For relative input names, the leading directory supplied to parse is the automatic bundle root, so ../examples/cmu/a.acmetest becomes cmu/a.acmetest.pt. Absolute input names use their common directory as the automatic root. The parse --base-directory DIR option selects the root explicitly. Artifact names always use / separators and must be relative, traversal-free paths. The input’s source extension is retained so bundle output has the same names as generated-target output (for example, a.g4.tree and a.g4.errors).

Parse-tree commands such as xpath, xquery, foldlit, rename, sort, unfold, unfoldlit, and combine replace the contents of the corresponding .pt members and pass .errors and other bundle members through unchanged. Their output is another PAX/tar bundle, so commands remain composable with |.

text and tree write PAX/tar bundles by default, replacing each .pt member with its rendered artifact while passing unrelated members through. Use --text for their human-readable stdout modes. Other display commands, such as xml, dot, and tokens, still write text by default and require --bundle to operate as bundle filters.

find examples -type f \
  | dotnet trash parse --allstar -L interp -x \
  | dotnet trash tree \
  | dotnet trash sponge -o pt-allstar -c

sponge detects the PAX/tar stream automatically. Under the directory supplied with -o, it reconstructs every .pt parse tree as source text using its recorded source filename and copies .errors and other artifacts unchanged. It rejects absolute paths, . or .. path components, links, duplicate destinations, and overwrites unless explicitly allowed.

For backward compatibility, commands also auto-detect and read the former JSON array representation of parsing result sets. Newly produced pipeline output is PAX/tar by default.

About

The source code for the extension is open source, free of charge, and free of ads. For the latest developments on the extension, check out my blog.

Building

git clone https://github.com/kaby76/Trash
cd Trash
make clean; make; make install

You must have the NET SDK version 10 installed to build and run.

Releases

See https://github.com/kaby76/Trash/releases.

If you have any questions, email me at ken.domino gmail.com