CAI NEURAL API 
Everyone says Pascal died in the 90s. Meanwhile, this Pascal library runs GPT-style transformers, Mixture-of-Experts, RWKV, xLSTM, spiking neurons, and normalizing flows, natively, with no Python nor CUDA lock-in. Your PyTorch project: 4GB of dependencies, a conda environment that breaks monthly. This: compile to a single native binary with AVX. This is deep learning the way systems programmers wish it worked.
And it actually runs the real thing. Point the ChatTerminal example at a HuggingFace instruct model — Qwen2.5 (0.5B up to 32B), Qwen3, Qwen3.8, Llama (TinyLlama, SmolLM2), Mistral 7B, Phi-3-mini or the OLMoE Mixture-of-Experts — and chat with a genuine LLM in your terminal, or expose it as an OpenAI-compatible server with ChatServer: OpenCL-accelerated, no Python, no CUDA — with more architectures landing all the time. Then hand a text prompt to MusicGenText and watch the same library turn “lo-fi hip-hop with a mellow piano” into actual audio — T5 encoder and MusicGen decoder. One native binary, from a 20-line CIFAR demo to a billion-parameter transformer.
CAI NEURAL API is a Pascal deep-learning library optimized for the AVX, AVX2 and AVX512 instruction sets and for OpenCL devices from AMD, Intel and NVIDIA. Tested on Windows and Linux.
Highlights
- 🗣️ Run real LLMs, natively — import a HuggingFace instruct checkpoint (Qwen2.5/Qwen3/Qwen3.8, Llama, Mistral, Phi-3, OLMoE) and chat from your terminal with
ChatTerminal, or serve the same checkpoint over an OpenAI-style HTTP endpoint with ChatServer, so any OpenAI client or SDK can call it. ChatServer’s POST /v1/systemone (TypeSafe/Jev typed decisions: a state and typed questions in, option probabilities out, through the AnyJev L0 readout, or through a Kev / Open-Jev LoRA checkpoint with --decision-adapter) supports Open-Jev-2B (user-tested on its Qwen/Qwen3.5-2B base with --int4 --gpu); AnyJev L0, Kev and the other Open-Jev sizes are planned (coded). No Python, no CUDA.
- 🎨 Text-to-image — turn a text prompt into a 1024x1024 image with
QwenImage (Qwen-Image-2.1: Qwen3-VL text encoder, 7B MMDiT transformer, VAE decoder; int8/int4 weights, OpenCL GPU by default).
- 🎵 Text-to-music — turn a text prompt into audio with
MusicGenText (T5 encoder + MusicGen decoder).
- 🔊 More audio — separate a song into stems with
MusicSourceSeparation, talk to an audio-aware model with Qwen2AudioChat, or turn English text into speech with KokoroTTS (Kokoro-82M; a pure Pascal G2P reads misaki’s dictionaries; REPL or one-shot).
- 🧠 Modern architectures, built in — GPT-style transformers, Mixture-of-Experts, RWKV, xLSTM, spiking neurons and normalizing flows, all as native Pascal layers.
- 🖼️ Image super-resolution — upscale images with a trained network via the
SuperResolution command-line tool (ships with a ready-to-use model).
- ⚡ Fast everywhere — AVX/AVX2/AVX512 CPU paths plus OpenCL offload for AMD, Intel and NVIDIA GPUs.
- 📦 Single-binary deploy — compile to one self-contained native executable. No runtime, no dependency tree.
- 📖 Readable at native speed — clean, easy-to-learn Pascal that compiles to fast native code. No trade-off between readable source and raw performance.
Prerequisites
You’ll need Lazarus development environment. If you have an OpenCL capable device, you’ll need its OpenCL drivers. Many examples use the CIFAR-10 dataset. You’ll also find examples for the CIFAR-100, MNIST, Fashion MNIST and the Places365-Standard Small images 256x256 dataset.
Will It Work with Delphi?
This project is Lazarus / Free Pascal based. The master branch does not work with Delphi. For Delphi, use v2.0.0 instead.
Installation
Clone this project, add the neural folder to your Lazarus unit search path and you’ll be ready to go!
Documentation
The documentation covers:
Intro Videos
 |
 |
 |
| Basics of Neural Networks in Pascal - Loading and Saving |
Neural Networks for Absolute Beginners! Learning a Simple Function |
Coding a Neural Network in Pascal that Learns to Calculate the Hypotenuse |
Easy Examples First Please!
You can click on the image above to watch the video.
Assuming that you would like to train a neural network to learn a function that has 2 inputs and one output, you could start with something like this:
NN.AddLayer([
TNNetInput.Create(2),
TNNetFullConnectReLU.Create(32),
TNNetFullConnectReLU.Create(32),
TNNetFullConnectLinear.Create(1)
]);
The example above has 2 inputs (TNNetInput), 2 dense layers (TNNetFullConnectReLU) with 32 neurons each and one output (TNNetFullConnectLinear).
You can learn more about how to build and train simple neural networks at the following source code examples:
Loading and Saving Neural Networks
Loading is very easy:
NN := TNNet.Create;
NN.LoadFromFile('MyTrainedNeuralNetwork.nn');
Saving is as easy:
NN.SaveToFile('MyTrainedNeuralNetwork.nn');
Simple Image Classification Examples
CIFAR-10 Image Classification Example
The CIFAR-10 dataset is a well-known collection of images commonly used to train machine learning and computer vision algorithms. It was created by the Canadian Institute for Advanced Research (CIFAR). It contains 60K 32x32 color images. The images are classified into 10 different classes, with 6,000 images per class. The classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Despite its relatively low resolution and small size, CIFAR-10 can be challenging for models to achieve high accuracy, making it a good dataset for testing advancements in machine learning techniques.
Follows a source code example for the CIFAR-10 image classification:
NN := TNNet.Create();
NN.AddLayer([
TNNetInput.Create(32, 32, 3), //32x32x3 Input Image
TNNetConvolutionReLU.Create({Features=}16, {FeatureSize=}5, {Padding=}0, {Stride=}1, {SuppressBias=}0),
TNNetMaxPool.Create({Size=}2),
TNNetConvolutionReLU.Create({Features=}32, {FeatureSize=}5, {Padding=}0, {Stride=}1, {SuppressBias=}0),
TNNetMaxPool.Create({Size=}2),
TNNetConvolutionReLU.Create({Features=}32, {FeatureSize=}5, {Padding=}0, {Stride=}1, {SuppressBias=}0),
TNNetFullConnectReLU.Create({Neurons=}32),
TNNetFullConnectLinear.Create(NumClasses),
TNNetSoftMax.Create()
]);
CreateCifar10Volumes(ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes);
WriteLn('Neural Network will minimize error with:');
WriteLn(' Layers: ', NN.CountLayers());
WriteLn(' Neurons:', NN.CountNeurons());
WriteLn(' Weights:', NN.CountWeights());
NeuralFit := TNeuralImageFit.Create;
NeuralFit.InitialLearningRate := fLearningRate;
NeuralFit.Inertia := fInertia;
NeuralFit.Fit(NN, ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes, NumClasses, {batchsize}128, {epochs}100);
These examples train a neural network to classify images in classes such as: image has a cat, image has a dog, image has an airplane…
You can save and load trained models (neural networks) with TNNet.SaveToFile and TNNet.LoadFromFile. The file format is portable meaning that you can train on CPU and run on GPU or train in AMD and run on ARM as examples. The following code shows a simple example for image classification loading a pre-trained model:
procedure ClassifyOneImageSimple;
var
NN: TNNet;
ImageFileName: string;
NeuralFit: TNeuralImageFit;
begin
WriteLn('Loading Neural Network...');
NN := TNNet.Create;
NN.LoadFromFile('SimplePlantLeafDisease-20230720.nn');
NeuralFit := TNeuralImageFit.Create;
ImageFileName := 'plant/Apple___Black_rot/image (1).JPG';
WriteLn('Processing image: ', ImageFileName);
WriteLn(
'The class of the image is: ',
NeuralFit.ClassifyImageFromFile(NN, ImageFileName)
);
NeuralFit.Free;
NN.Free;
end;
Youtube Videos
|
|
|
| Basics of Neural Networks in Pascal - Loading and Saving |
Neural Networks for Absolute Beginners! Learning a Simple Function |
Coding a Neural Network in Pascal that Learns to Calculate the Hypotenuse |
 |
 |
 |
| Pre-trained Neural Networks & Transfer Learning with Pascal’s CAI Neural API |
Coding a Neural Network in Pascal that Learns the OR Boolean Operation |
A Dive into Identity Shortcut Connection - The ResNet building block |
 |
 |
 |
| Increasing Image Resolution with Neural Networks |
Ultra Fast Single Precision Floating Point Computing |
AVX and AVX2 Code Optimization |
Some videos make referrence to uvolume unit. The current neuralvolume unit used to be called uvolume. This is why
it’s mentioned.
Advanced Examples
Although these examples require deeper understanding about neural networks, they are very interesting:
There is also a full set of examples that you can look at.
Volumes
Volumes behave like dynamically created arrays. They are the main array like structure used by this API. TNNetVolume class allows you to create volumes that can be accessed as 1D, 2D or 3D arrays and be operated with Advanced Vector Extensions (AVX) - Single Instruction Multiple Data (SIMD) instruction set. The usual way to create a volume is:
constructor Create(pSizeX, pSizeY, pDepth: integer; c: T = 0);
You can access the data as 1D or 3D with:
property Raw[x: integer]: T read GetRaw write SetRaw;
property Data[x, y, d: integer]: T read Get write Store; default;
Your code will look like this:
// Usage Examples
vInput := TNNetVolume.Create(32, 32, 3);
vInput[1, 1, 1] := 1;
vInput[2, 2, 2] := vInput[1, 1, 1] + 1;
vInput.Raw[10] := 5;
vInput.RandomizeGaussian();
WriteLn('Avg: ', vInput.GetAvg());
WriteLn('Variance: ', vInput.GetVariance());
WriteLn('Std Dev: ', vInput.GetStdDeviation());
WriteLn('Multiplying by 10');
vInput.Mul(10);
WriteLn('Avg: ', vInput.GetAvg());
WriteLn('Variance: ', vInput.GetVariance());
WriteLn('Std Dev: ', vInput.GetStdDeviation());
As examples, you can add, subtract, multiply and calculate dot products with:
procedure Add(Original: TNNetVolume); overload;
procedure Sub(Original: TNNetVolume); overload;
procedure Mul(Value: Single); overload;
function DotProduct(Original: TNNetVolume): TNeuralFloat; overload;
In the case that you need the raw position or raw pointer to an element of the volume, you can get with:
function GetRawPos(x, y, d: integer): integer; overload;
function GetRawPos(x, y: integer): integer; overload;
function GetRawPtr(x, y, d: integer): pointer; overload;
function GetRawPtr(x, y: integer): pointer; overload;
function GetRawPtr(x: integer): pointer; overload;
You can easily operate volumes with OpenCL via TEasyOpenCLV:
TEasyOpenCLV = class (TEasyOpenCL)
public
function CreateBuffer(flags: cl_mem_flags; V: TNNetVolume): cl_mem; overload;
function CreateInputBuffer(V: TNNetVolume): cl_mem; overload;
function CreateHostInputBuffer(V: TNNetVolume): cl_mem; overload;
function CreateOutputBuffer(V: TNNetVolume): cl_mem; overload;
function CreateBuffer(V: TNNetVolume): cl_mem; overload;
function WriteBuffer(buffer: cl_mem; V: TNNetVolume; blocking: cl_bool = CL_FALSE): integer;
function ReadBuffer(buffer: cl_mem; V: TNNetVolume; blocking: cl_bool = CL_TRUE): integer;
function CreateAndWriteBuffer(V: TNNetVolume; var buffer: cl_mem): integer; overload;
function CreateAndWriteBuffer(V: TNNetVolume): cl_mem; overload;
function CreateWriteSetArgument(V: TNNetVolume; kernel:cl_kernel; arg_index: cl_uint): cl_mem;
function CreateOutputSetArgument(V: TNNetVolume; kernel:cl_kernel; arg_index: cl_uint): cl_mem;
end;
Volume Pairs, Volume Lists and Volume Pair Lists
Volumes can be organized in pairs:
/// Implements a pair of volumes
TNNetVolumePair = class(TObject)
protected
FA: TNNetVolume;
FB: TNNetVolume;
public
constructor Create(); overload;
constructor Create(pA, pB: TNNetVolume); overload;
constructor CreateCopying(pA, pB: TNNetVolume); overload;
destructor Destroy(); override;
property A:TNNetVolume read FA;
property B:TNNetVolume read FB;
property I:TNNetVolume read FA;
property O:TNNetVolume read FB;
end;
Depending on the problem that you are trying to solve, modelling the training with pairs or pair lists might be helpful. Typically, a pair will be (input, desired output).
This is how volume lists and volume pair lists have been implemented:
TNNetVolumeList = class (specialize TFPGObjectList<TNNetVolume>
TNNetVolumePairList = class (specialize TFPGObjectList<TNNetVolumePair>)
Neural Network Layers
The layered structure of artificial neural networks is inspired by the organization of the human brain and nervous system. In the human brain, information processing occurs in a hierarchical manner. Sensory inputs are first processed by lower-level neurons, which extract simple features. These features are then passed on to deeper neurons that combine them to recognize more complex patterns. This hierarchical processing is mirrored in artificial neural networks through the use of stacked layers.
Biological neurons are connected to each other through synapses, forming complex networks. Similarly, in artificial neural networks, neurons in one layer are connected to neurons in the next layer, mimicking this interconnected structure. Biological neurons fire (activate) based on a non-linear response to their inputs. This non-linearity is crucial for the brain’s ability to learn complex patterns. In artificial neural networks, we use non-linear activation functions (such as ReLU) to introduce this non-linearity. Different regions of the brain specialize in processing different types of information. For instance, the visual cortex has layers specialized for detecting edges, shapes, and complex objects. This specialization is reflected in artificial neural networks, where different layers can learn to recognize different levels of abstraction.
In the context of artificial neural networks, we can see this biologically-inspired layered approach implemented. For example:
NN := TNNet.Create();
NN.AddLayer([
TNNetInput.Create(32, 32, 3),
TNNetConvolutionLinear.Create({neurons=}16, {featuresize}3, {padding}1, {stride}1),
TNNetReLU6.Create()
]);
This code snippet demonstrates the creation of a neural network with an input layer and a convolutional layer followed by a ReLU6 activation. This structure is inspired by the visual cortex in the brain, where neurons respond to specific patterns in their receptive fields, similar to how convolutional layers operate. The CAI Neural API also supports the creation of more complex, biologically-inspired architectures. These architectures are designed with multiple layers of different types, mirroring the complex structure of the brain.
Artificial neural networks with multiple layers and specialized structures are inspired by the hierarchical and specialized nature of biological neural processing. It’s important to note that while artificial neural networks are inspired by biological neural networks, they are highly simplified models. The human brain is far more complex, with various types of neurons, complex connectivity patterns, and mechanisms we don’t yet fully understand. However, the layered structure in artificial neural networks has proven to be a powerful approach for solving complex problems in machine learning, inspired by the remarkable capabilities of biological neural networks.
The input layer serves as the gateway to the entire network. It’s like the sensory organs of our brain, receiving information from the outside world. Without an input layer, the neural network would have no way to receive and interpret the initial data, making it impossible to perform any meaningful computations or learning tasks. The TNNetInput class implements the input layer.
Fully Connected (Dense) Layers
Fully connected layers, also known as dense layers, are a fundamental component of neural networks. In these layers, every neuron is connected to every neuron in the previous layer, allowing for comprehensive information processing across the entire network.
In the context of the CAI Neural API, fully connected layers are represented by various classes derived from TNNetFullConnect. These layers play a crucial role in transforming input data and learning complex patterns.
The computation process in a fully connected layer involves:
- Multiplying input values by the layer’s weights.
- Adding bias terms (if not suppressed).
- Applying an activation function (if present).
Key types of fully connected layers include:
TNNetFullConnectLinear: a basic fully connected layer without an activation function. It performs a linear transformation of the input data.TNNetFullConnectReLU: incorporates the Rectified Linear Unit (ReLU) activation function. ReLU introduces non-linearity by outputting the input for positive values and zero for negative values, helping the network learn complex patterns.TNNetFullConnectSigmoid: applies the sigmoid activation function to the layer’s output. Sigmoid squashes the output between 0 and 1, useful for binary classification tasks.TNNetComplexLinear: the 2-dimensional complex base rung of the same Cayley–Dickson family — each 2-channel group (Re,Im) is multiplied by a learned complex w = a + b·i via the 2×2 complex-multiply block [[a,-b],[b,a]] (Re' = a·Re − b·Im, Im' = a·Im + b·Re), using ~1/2 the weights of an equal-width real dense layer while still mixing the real and imaginary parts. The forward block equals the true complex product and is norm-multiplicative |w·X| = |w|·|X|. Input/output Depth must be multiples of 2. See examples/ComplexLinear/.TNNetQuaternionLinear: a hypercomplex dense layer that shares each 4×4 Hamilton block from a single learned quaternion, coupling 4-channel groups (rotation/scaling in quaternion space) with ~1/4 the weights of an equal-width real dense layer. Input/output Depth must be multiples of 4.TNNetOctonionLinear: the 8-dimensional octonion (Cayley–Dickson) generalization of the above — each 8-channel group is multiplied by a learned octonion via the (non-associative) octonion product, assembled as an 8×8 signed block, using ~1/8 the weights of an equal-width real dense layer. The implementation hard-codes an auditable Cayley–Dickson sign table verified by a norm-multiplicativity test (|W·X| = |W|·|X|). Input/output Depth must be multiples of 8. See examples/OctonionLinear/.
Fully connected layers are typically used in neural network architectures as:
- Hidden layers for processing and transforming features.
- Output layers for producing final predictions.
For example, in the provided context, we see a simple neural network structure:
NN := TNNet.Create();
NN.AddLayer([
TNNetInput.Create(2), // Input layer with 2 inputs
TNNetFullConnect.Create(2), // Hidden fully connected layer with 2 neurons
TNNetFullConnect.Create(1) // Output fully connected layer with 1 neuron
]);
| Layer Name |
Input/Output Dimensions |
Activation |
Description |
TNNetFullConnectLinear |
1D, 2D, or 3D |
None |
Fully connected layer without an activation function (linear). |
TNNetFullConnect |
1D, 2D, or 3D |
tanh |
Fully connected layer with tanh as the default activation function. |
TNNetFullConnectReLU |
1D, 2D, or 3D |
ReLU |
Fully connected layer with ReLU activation. |
TNNetFullConnectSigmoid |
1D, 2D, or 3D |
Sigmoid |
Fully connected layer with Sigmoid activation. |
TNNet.AddGroupedFullConnect |
1D, 2D, or 3D |
Optional |
Adds a grouped fully connected layer, inspired by TNNet.AddGroupedConvolution. |
TNNetBitLinear |
1D, 2D, or 3D |
None |
BitNet b1.58 ternary-weight linear layer: forward uses per-neuron absmean-quantized weights Wq = scale*round(clip(W/scale,-1,+1)) with `scale = mean( |
TNNetCirculantLinear |
1D, 2D, or 3D (n=Depth) |
None |
Structured-matrix dense layer whose n×n weight matrix is CIRCULANT — every row is a cyclic shift of one learned length-n kernel c, so it stores O(n) weights instead of O(n²) and the map is y = circular_convolution(c, x) (+ bias). Default forward/backward is the exact direct O(n²) circular sum; set UseFFT := true for an opt-in O(n log n) radix-2 FFT fast path (power-of-two n; default OFF, bit-for-bit equivalent to the direct path to <1e-5). Distinct from LoRA (low-rank), AddGroupedFullConnect (block-diagonal) and TNNetBitLinear (quantized dense). See examples/CirculantLinear/. |
TNNetHouseholderLinear |
1D, 2D, or 3D (n=Depth) |
None |
EXACTLY-orthogonal dense layer whose n×n weight is parameterized as a product of K Householder reflections Q = H_1·H_2·…·H_K with H_i = I − 2·(v_i·v_iᵀ)/(v_iᵀ·v_i); the trainable parameters are the K reflection vectors v_i ∈ ℝ^n. Q is orthogonal for any v_i (no constrained optimization or re-projection), so the map is exactly norm/volume preserving (‖y‖ = ‖x‖ with bias off) and exactly invertible (Qᵀ = H_K·…·H_1). Forward applies the reflections one at a time (O(K·n) per sample, never materializing Q), y = Q·x (+ bias); backward caches each intermediate x_i = H_i·…·H_K·x for a closed-form per-v_i gradient and propagates dL/dx = Qᵀ·dL/dy. An exactly-orthogonal Jacobian is an isometry, so deep plain stacks neither explode nor vanish (the building block for orthogonal/unitary RNNs and reversible/normalizing-flow nets). Distinct from TNNetSpectralNorm (bounds only σ_1, leaves the other singular values free), TNNetCirculantLinear/Toeplitz (constrain the matrix form, not orthogonality) and Muon (orthogonalizes the update, not the weight). The v_iᵀ·v_i → 0 denominator is guarded (degenerate reflection ≈ identity). Composite helper TNNet.AddHouseholderLinear(N, NumReflections, UseBias) (default K = n for a full orthogonal group element; K < n gives a cheaper sub-group). See examples/HouseholderOrthogonal/. |
TNNetHyperbolicLinear |
1D, 2D, or 3D (Depth-vector point in the ball) |
None |
Poincaré-ball HYPERBOLIC dense layer (Ganea, Bécigneul & Hofmann 2018, Hyperbolic Neural Networks; Nickel & Kiela 2017): treats the Depth-vector input x as a point inside the open Poincaré ball of radius 1/√c (fixed curvature c>0, default 1.0) and computes the Möbius matrix–vector product followed by a hyperbolic (Möbius) bias translation, y = exp_0(M·log_0(x)) ⊕_c b, where log_0(x)=(1/√c)·atanh(√c‖x‖)·x/‖x‖ and exp_0(v)=(1/√c)·tanh(√c‖v‖)·v/‖v‖ are the log/exp maps at the origin and a ⊕_c b is Möbius addition. Reuses the TNNetFullConnectLinear weight layout (one neuron per output dim holds matrix row M[j]; the Möbius bias b is held coordinate-wise in the per-neuron bias). Genuinely non-Euclidean — unlike every other dense layer here (all flat sum_j W·x plus optional Euclidean bias), the matmul and bias act in hyperbolic space, the natural geometry for embedding trees / hierarchies with low distortion; the output always stays inside the ball (‖y‖<1/√c). Backward is the EXACT analytic chain rule through both radial atanh/tanh maps and the Möbius-addition Jacobian (input, matrix-row and Möbius-bias gradients all numerically gradient-checked); the ‖x‖→1/√c boundary and ‖x‖→0 origin are guarded by an EPS/series fallback so the Jacobian stays finite. Curvature c round-trips via FFloatSt[0]; suppress-bias via FStruct[3]. Created with TNNetHyperbolicLinear.Create(Dout) (default c=1) or Create(Dout, c). Optional trainable curvature: Create(Dout, c, SuppressBias, LearnCurvature:=true) makes c a single learnable scalar via the constrained-scalar pattern — one extra 1-weight neuron holds a raw value mapped through c = 0.01 + 3.99*sigmoid(raw) (kept strictly positive/bounded), the exact dL/draw is accumulated via a forward-mode tangent of y through the log_0/exp_0/Möbius-add chain, and the flag round-trips via FStruct[4]. The default (fixed-c) path is byte-for-byte unchanged. |
TNNetHyperbolicDistance |
1D, 2D, or 3D (Depth-vector point in the ball) |
None |
Poincaré-ball distance READOUT head (companion to TNNetHyperbolicLinear): maps the Depth-vector input x (a point inside the curvature-c Poincaré ball) to a K-vector of hyperbolic distances to K learnable prototype points p_k, d_k = dist_c(x, p_k) = (2/√c)·atanh(√c·‖(-x) ⊕_c p_k‖) (Möbius addition ⊕_c as in TNNetHyperbolicLinear). Reuses the TNNetFullConnectLinear weight layout (one neuron per prototype holds p_k, bias suppressed). A usable hyperbolic classification/regression head — small distances mean “close in hyperbolic space”. Backward is the EXACT analytic chain rule through the radial atanh and the Möbius-addition Jacobian (input and per-prototype gradients both numerically gradient-checked); boundary/origin guarded by EPS/series fallback. Curvature c round-trips via FFloatSt[0], prototype count K via FStruct[0]. Created with TNNetHyperbolicDistance.Create(K) (default c=1) or Create(K, c). See examples/HyperbolicEmbedding/. |
TNNetMonarchLinear |
1D, 2D, or 3D (n=Depth) |
None |
Sub-quadratic STRUCTURED dense layer whose n×n weight is a MONARCH matrix (Dao et al. 2022, Monarch: Expressive Structured Matrices for Efficient and Accurate Training): a product of two block-diagonal factors interleaved by a fixed reshape-transpose permutation, y = Pᵀ·(L·(P·(R·x))) (+ bias), where R and L are block-diagonal with b blocks of size m×m (n = b·m, default b = round(√n) for perfect-square n, else the largest divisor ≤ √n; b round-trips via FStruct[1]) and P is the fixed (b,m)→(m,b) index-gather permutation (no weights). Stores/runs in O(n^1.5) instead of O(n²) yet provably contains the DFT, the Hadamard transform and ordinary convolutions — a genuinely different structured operator from TNNetCirculantLinear (single cyclic kernel), LoRA (low-rank) and AddGroupedFullConnect (one block-diagonal, no permutation mix). Forward applies R, permute, L, un-permute as four cheap passes (never materializing the dense n×n); backward is the exact transpose chain (dL/dx, dL/dR, dL/dL all block-local matmuls), both factor gradients numerically gradient-checked. Suppress-bias via FStruct[3]. The map is square, so n is inferred from the previous layer’s size; created with TNNetMonarchLinear.Create() (or Create(1) to suppress bias). |
TNNetKroneckerLinear |
1D, 2D, or 3D (n=Depth) |
None |
Sub-quadratic STRUCTURED dense layer whose n×n weight is a single KRONECKER PRODUCT W = A ⊗ B of two small learned factors A (p×p) and B (q×q) with n = p·q. Stores only O(p²+q²) ≈ O(n) weights instead of O(n²), and the dense n×n Kronecker matrix is NEVER materialized: x is reshaped to a q×p matrix X (X[i,j] = x[i·p+j]) and the matvec is two small GEMMs Y = B·X·Aᵀ (O(n·(p+q)) = O(n^1.5)), flattened back as y[i·p+j] = Y[i,j] (+ bias). Under this row-major vec convention y = vec(B·X·Aᵀ) exactly equals (A⊗B)·x. Backward is the exact transpose chain dL/dX = Bᵀ·dY·A, dL/dA = dYᵀ·(B·X), dL/dB = dY·(X·Aᵀ)ᵀ (all small GEMMs; input, dA and dB all numerically gradient-checked). The factor split p defaults to round(√n) (largest divisor ≤ √n when n is not a perfect square) and round-trips via FStruct[1]; q = n div p. A genuinely different structured operator from TNNetCirculantLinear (single cyclic kernel), TNNetHouseholderLinear (exactly orthogonal), TNNetMonarchLinear (two block-diagonal factors + permutation) and LoRA (low-rank): Kronecker is a single tensor-product factorisation. Suppress-bias via FStruct[3]. The map is square, so n is inferred from the previous layer’s size; created with TNNetKroneckerLinear.Create() (or Create(1) to suppress bias, or Create(0, p) to pin the factor split). See examples/KroneckerLinear/. |
TNNetTropicalLinear |
1D, 2D, or 3D (Din=Depth) |
None |
A max-plus / min-plus morphological dense layer that computes in the TROPICAL (max-plus) semiring instead of the usual multiply-accumulate ring: y_i = max_j (x_j + W[i,j]) (a morphological DILATION), with a paired ERODE mode y_i = min_j (x_j + W[i,j]) selected by a constructor flag (round-trips via FStruct[6]). The weights are learnable additive thresholds and the combine op is max/min, so the layer learns piecewise-linear convex (dilation) / concave (erosion) functions and tropical polynomials — a genuinely different operator from TNNetFullConnect* and the structured-linear family (TNNetCirculantLinear/TNNetHouseholderLinear/TNNetBitLinear, all sum_j W·x) and from parameter-free max/min pooling. Forward is O(Din·Dout); backward is the same hard arg-max/arg-min subgradient as TNNetMaxPool — cache the winning j* per output and route dL/dx[j*] += dy_i, dL/dW[i,j*] += dy_i (non-differentiable at ties). Both the input and weight gradients are numerically gradient-checked (away from the tie kink). Created with TNNetTropicalLinear.Create(Dout) (dilation) or Create(Dout, 1) (erode). See examples/TropicalMorphology/. |
TNNetTropicalConv |
2D or 3D (SizeX x SizeY x Depth) |
None |
The SPATIAL sibling of TNNetTropicalLinear — a grayscale morphological dilation/erosion convolution with a learnable additive structuring element (SE) over a (SizeX,SizeY,Depth) patch: y[x,y,co] = max_{dx,dy,ci} (input[x+dx,y+dy,ci] + SE[dx,dy,ci,co]) (dilation), with a paired ERODE mode min_{dx,dy,ci} (input − SE) selected by a constructor flag (round-trips via FStruct[6]). Subclasses TNNetConvolutionLinear for the conv geometry (kernel/padding/stride) but replaces the multiply-accumulate with the max-plus/min-plus semiring and a single SE weight neuron — distinct from parameter-free TNNetMaxPool (learnable additive SE, not a fixed window) and from ordinary linear conv (sum W·x). Backward is the hard arg-max/arg-min one-hot subgradient (MaxPool convention): cache the winning (dx,dy,ci) tap per output cell and route dy to that single input cell and that single SE tap (non-differentiable at ties). Both input and SE gradients numerically gradient-checked (away from the tie kink). Created with TNNetTropicalConv.Create(Features, FeatureSize, Padding, Stride) (dilation) or Create(Features, FeatureSize, Padding, Stride, 1) (erode). |
TNNetCondConv |
2D or 3D (SizeX x SizeY x Depth) |
None |
Conditionally-parameterized (“dynamic”) convolution (Yang et al. 2019, CondConv, NeurIPS): owns a bank of K expert kernels W_1..W_K (each a normal Features × FeatureSize × FeatureSize × InChannels kernel) plus a tiny per-sample routing head (global-average-pool → FullConnect → sigmoid) emitting K mixing coefficients alpha_k per input sample; the effective kernel is the per-sample blend W_eff = sum_k alpha_k·W_k applied as ONE ordinary convolution — so inference cost stays that of a single conv regardless of K while capacity grows with the bank. Backward routes dL/dW_k = alpha_k·dL/dW_eff, sends dL/dalpha_k = <dL/dW_eff, W_k> back through the sigmoid + FC + pool into the input, and propagates the standard conv input gradient through W_eff (input, expert-bank, and routing-head gradients all numerically gradient-checked). Distinct from TNNetHyperConv (GENERATES the whole kernel from a second tensor in one shot) and AddMixtureOfExperts (mixes K expert OUTPUTS post-hoc, K forward passes); CondConv mixes K kernels BEFORE the conv (one forward pass). K/Features/FeatureSize/Padding/Stride round-trip via FStruct. Created with TNNetCondConv.Create(K, Features, FeatureSize, Padding, Stride). See examples/CondConv/. |
TNNetSpectralNorm |
1D, 2D, or 3D |
None |
Spectral-normalized dense layer (Miyato et al. 2018): forward divides the weight matrix by its largest singular value sigma_1 (estimated by power iteration, Iters steps in FStruct[5], default 10) so the effective operator W/sigma_1 has spectral norm ~1; sigma_1 is treated as constant in backward (input error propagated through the scaled weights). |
TNNetHighway |
1D, 2D, or 3D (shape-preserving over Depth) |
Optional |
Highway-network layer (Srivastava, Greff & Schmidhuber 2015): y = T(x)⊙H(x) + (1−T(x))⊙x with an input-dependent transform gate T(x) = sigmoid(W_T·x + b_T) and learned transform H(x) = activation(W_H·x + b_H). The input-dependent learned-gate ancestor of the ResNet skip connection — unlike TNNetReZero (scalar) and TNNetGatedResidual (per-channel constant), the gate is computed from the activation each forward pass and carries the identity (1−T)·x. Gate bias inits negative (b_T ≈ −1.5, the paper’s carry trick) so a fresh deep stack starts near identity. Per-channel pointwise, so it composes inside the residual builders. See examples/HighwayDepth/. |
TNNetHyperLinear |
main: 1D/2D/3D → Dout |
None |
HyperNetwork dense layer (Ha, Dai & Le 2016): owns NO trainable weights — its weight matrix is GENERATED by an upstream net and read from a SECOND input tensor (WeightsSource, a flat Din*Dout (+Dout) row-major matrix + optional bias) rather than from Neurons[]. Two-source wiring like TNNetCrossAttention. Forward y = W_gen·x (+b); backward propagates into BOTH the main input (W_gen^T·dy) and the generated-weights tensor (dy⊗x, dy) so the generator trains end-to-end. Composite helper TNNet.AddHyperLinear(Din, Dout, ContextLayer, UseBias=true) wires the generator FullConnect (off ContextLayer) and the weightless TNNetHyperLinear (off the main path) in one call. See examples/HyperNetwork/. |
TNNetHyperConv |
main: 3D image → VALID conv |
None |
Spatial cousin of TNNetHyperLinear — a weightless convolution whose kernel is GENERATED by an upstream net and read from a SECOND input tensor instead of Neurons[]. VALID conv, stride 1; kernel laid out W[o,ky,kx,i] flat (((o*K+ky)*K+kx)*InChannels + i) plus optional per-output-channel bias. Backward propagates into BOTH the main image and the generated-kernel tensor so the generator trains end-to-end. Composite helper TNNet.AddHyperConv(InChannels, OutChannels, FeatureSize, ContextLayer, UseBias=true) wires the generator FullConnect and the weightless TNNetHyperConv in one call. Note: the generator emits the whole OutC*K*K*InC kernel in one shot, which caps kernel/channel size (memory/param trade-off). |
TNNetModulatedConv2D |
main: 3D image → SAME conv |
Conv kernel (+bias) |
StyleGAN2 weight-modulated / demodulated convolution (Karras et al. 2020, Analyzing and Improving the Image Quality of StyleGAN) — the style-synthesis PRIMITIVE, a WEIGHT-SPACE op (not an activation). Unlike TNNetHyperConv (whose whole kernel is generated upstream) this OWNS a trainable base kernel in Neurons[] and per forward pass SCALES it by a per-input-channel style vector read from a SECOND input then RENORMALIZES: modulate w'[o,k,i]=s_i·w[o,k,i], demodulate w''[o,k,i]=w'[o,k,i]/sqrt(Σ_{k,i} w'^2 + ε) (toggle off for the toRGB 1×1 conv). SAME padding, stride 1; backward routes gradient to the owned kernel (through the demod normalization, dd/dw'=-d^3·w'), the feature input, AND the style branch (so the mapping/affine trains end-to-end). Two-source wiring like TNNetHyperConv (Create(NumFeatures, FeatureSize, SuppressBias, Demodulate, StyleSource)); FStruct + the injected style-source index round-trip through SaveToString. Used by BuildStyleGAN2Generator (neuralpretrained.pas). See examples/StyleGAN2Generate/. |
TNNetModernHopfield |
2D (SeqLen x 1 x d) |
Pattern bank |
Modern continuous Hopfield associative-memory layer (Ramsauer et al. 2020, Hopfield Networks is All You Need): an ENERGY-BASED memory that ITERATES a softmax retrieval to a fixed point against a learnable stored-pattern bank X of shape (NumPatterns, d). Per query position, xi := X^T·softmax(beta·X·xi) is applied for K update steps — K=1 reduces to ordinary attention, K>1 sharpens toward the nearest stored memory (the whole point). Inverse-temperature beta controls the metastable-vs-sharp regime. Forward caches per-step softmax weights; backward differentiates through the unrolled K steps (the SDPA softmax-Jacobian path, summed over steps, scattered into both bank and input). Composite helper TNNet.AddModernHopfieldRetrieval(NumPatterns, KSteps, Beta) wires a (SeqLen,1,d) query against a fresh learnable bank. See examples/HopfieldAssociativeMemory/. |
TNNetProductKeyMemory |
Input: SeqLen x 1 x QueryDim (even QueryDim, divisible by Heads with QueryDim/Heads even). Output: SeqLen x 1 x (Heads*ValueDim). |
Half-keys + values (per head) |
Large, sparsely-accessed product-key memory (Lample et al. 2019, Large Memory Layers with Product Keys): factorizes a NumKeys-row memory into the Cartesian product of two small half-key banks K1, K2 (each sqrt(NumKeys) rows of (QueryDim/Heads)/2). A query is split in half, each half scored against its own bank, the top-TopK per half taken, the TopK x TopK combinations re-scored, and the global top-TopK product keys selected in O(sqrt(NumKeys)) work; a softmax over those gates a sparse weighted sum over only the touched rows of the learned value table. Multi-head (Heads>1): the query is split into Heads contiguous sub-queries, each running an independent product-key lookup against its own K1/K2/V banks (3*Heads neurons; head h owns neurons 3h,3h+1,3h+2), and the Heads ValueDim-wide outputs are concatenated along Depth. Heads=1 is byte-for-byte the v1 single-head path. Forward caches the chosen indices + weights; backward scatters the value-gradient into only the touched value rows and pushes the exact softmax-Jacobian score-gradient back through both half-key dot products into the query and K1/K2. Distinct from TNNetModernHopfield (dense softmax over a small fully-retrieved bank), TNNetEmbedding (one-hot lookup), and MoE (expert MLPs). Composite helper TNNet.AddProductKeyMemory(NumKeys, ValueDim, TopK, Heads). See examples/ProductKeyMemory/. |
TNNetNTMMemory |
Input: T x 1 x InputDim. Output: T x 1 x SlotWidth (per-step read vectors). |
Key/beta/erase/add projections |
The first writable differentiable external-memory layer in the library — a Neural Turing Machine (Graves et al. 2014, Neural Turing Machines, arXiv:1410.5401). Unlike the read-only associative memories (TNNetModernHopfield iterated recall, TNNetProductKeyMemory sparse lookup) it carries a persistent memory matrix M (NumSlots × SlotWidth) that the layer both reads and writes while sweeping the time axis. Per step the input is projected to a content key (cosine-addressed against every slot row, sharpened by a softplus key-strength beta, softmaxed over slots → weights w), a read r_t = w^T·M (the step’s output), and a sigmoid erase e + add a that update M[i] := M[i]·(1 − w[i]·e) + w[i]·a (erase-then-add, same w). The four projection matrices (8 neurons Wk,bk,Wb,bb,We,be,Wa,ba) are the only trainables; backward is full BPTT through the recurrent M update — because M_t depends on both M_{t-1} and w_t, dL/dM and dL/dw both chain backward across steps (input, read-path and write-projection gradients all numerically gradient-checked). NumSlots/SlotWidth round-trip via FStruct[0..1], InitVal via FFloatSt[0]; M is re-initialised to a small constant each sweep and is not persisted across loads (re-init on load, like SetAdjacency). v1 is content addressing only, single read+write head; location-based shift/interpolation addressing and the DNC temporal-link matrix are documented follow-ups. Created with TNNetNTMMemory.Create(NumSlots, SlotWidth, InitVal=0.001). See examples/NeuralTuringMachine/. |
TNNetSoftDecisionTree |
Input: flat Din-vector. Output: 1 x 1 x OutputDepth. |
None (mixture of leaf vectors) |
Differentiable soft (oblique) decision tree (Kontschieder et al. 2015, Deep Neural Decision Forests; Frosst & Hinton 2017, Distilling a Neural Network Into a Soft Decision Tree) — a structurally new hierarchical soft-routing paradigm for this library (not a matrix factorization, attention, recurrence or kernel method). A balanced binary tree of fixed depth D has 2^D−1 inner nodes and 2^D leaves. Each inner node i is a learnable linear gate p_i = sigmoid(beta·(w_i·x + b_i)); a sample reaches a leaf with probability equal to the product of the left (p_i) / right (1−p_i) gate decisions along its root-to-leaf path, and the output is the path-probability-weighted mixture `y = sum_l P(leaf=l |
TNNetGridSample |
main: 3D feature map SizeX x SizeY x Depth; second source: dense SizeX x SizeY x 2 normalized grid (x_n, y_n) in [-1,1] (same spatial size). Output: same shape as the features. |
None (pure differentiable resampling) |
Faithful port of torch.nn.functional.grid_sample (4-D case) — a two-source layer that warps the feature map by an EXPLICIT per-output-pixel sampling grid. Unlike TNNetAffineGridSample (grid from one global 2×3 theta) or TNNetFlowWarp/TNNetBackwardWarp (PIXEL-unit (dx,dy) flow), the grid here gives normalized [-1,1] sample coordinates over the output grid, exactly the torch convention. Supports mode='bilinear' (gsiBilinear) / 'nearest' (gsiNearest), padding_mode='zeros' (gspZeros) / 'border' (gspBorder), and the align_corners flag (all matched to torch: align_corners=True ⇒ px=(x_n+1)(W-1)/2, False ⇒ px=((x_n+1)W-1)/2). The bilinear forward delegates to the shared depth-column AVX bilinear-gather helper (NeuralBilinearGatherColumn, also used by TNNetAffineGridSample). Backward writes BOTH gradients — (a) dL/d(features) (scatter to the four bilinear neighbours; nearest scatters to the single rounded neighbour) and (b) dL/d(grid) (d(sampled)/d(px,py) chained through d(px)/d(x_n); nearest is locally constant ⇒ grid gradient 0; border clamp drops the saturated axis), both numerically gradient-checked. Two-source wiring like TNNetCrossAttention/TNNetAffineGridSample; the grid-source index plus the interp/pad/align_corners flags round-trip through SaveToString/LoadFromString. No trainable parameters. Created with TNNetGridSample.Create(GridSource, Interp, Pad, AlignCorners). |
TNNetFlowWarp |
main: 3D image SizeX x SizeY x Depth; second source: dense SizeX x SizeY x 2 (dx, dy) flow field (same spatial size). Output: same shape as the features. |
None (pure differentiable resampling) |
Dense per-pixel backward-warp sampler — the dense-field sibling of the PARAMETRIC TNNetAffineGridSample (one global 2×3 theta) and of TNNetDeformableConv’s per-tap bilinear sampler. Given a feature map and a two-channel (dx, dy) DENSE flow field of the SAME spatial size (a SECOND input source, wired like TNNetCrossAttention/TNNetAffineGridSample), produces the bilinearly-sampled backward-warp out(x,y,c) = in(x+dx(x,y), y+dy(x,y), c) with border-clamp edge handling (edge pixels replicated). Every output pixel carries its OWN learned displacement — exactly what optical-flow warping (RAFT) and RIFE/FILM frame interpolation / video stabilization need. Backward writes BOTH gradients: (a) dL/d(features) (the bilinear weights scatter the output gradient back onto the four clamped source neighbours, like TNNetAffineGridSample/TNNetDeformableConv) and (b) dL/d(flow) (the bilinear weights are differentiable in dx,dy via d(sampled)/d(sx), d(sampled)/d(sy), with d(sx)/d(dx)=d(sy)/d(dy)=1), both numerically gradient-checked. No trainable parameters; the injected flow-source layer index round-trips through SaveToString/LoadFromString. Created with TNNetFlowWarp.Create(FlowSource). See examples/FrameInterpolation/. |
TNNetBackwardWarp |
main: 3D image SizeX x SizeY x Depth; second source: dense SizeX x SizeY x 2 (dx, dy) flow field (same spatial size). Output: same shape as the image. |
None (pure differentiable resampling) |
RIFE-convention differentiable backward (inverse) warp — the named, convention-pinned resampler the RIFE frame-interpolation importer (BuildRIFEFromSafeTensors) wires against. Given an image and a two-channel (dx, dy) DENSE flow field of the SAME spatial size (a SECOND input source, wired like TNNetFlowWarp/TNNetCrossAttention), produces the bilinear backward-warp out(x,y,c) = image(x+dx(x,y), y+dy(x,y), c) with border-clamp padding. This pins RIFE’s torch convention exactly: flow in PIXEL units, F.grid_sample(mode='bilinear', padding_mode='border', align_corners=True) (which on integer pixels reduces to the pixel-space gather above). A sibling of TNNetFlowWarp (same math, different name/role). Backward writes BOTH gradients: (a) dL/d(image) (bilinear weights scatter the output gradient onto the four clamped source neighbours) and (b) dL/d(flow) (d(sampled)/d(sx), d(sampled)/d(sy), with d(sx)/d(dx)=d(sy)/d(dy)=1), both numerically gradient-checked. No trainable parameters; the injected flow-source layer index round-trips through SaveToString/LoadFromString. Created with TNNetBackwardWarp.Create(FlowSource). See examples/VideoFrameInterpolation/. |
TNNetRoIAlign |
main: 3D feature map SizeX x SizeY x Depth. Output: fixed PooledW x PooledH x Depth (channels preserved). |
None (parameter-free pooling) |
RoI Align (He et al. 2017, Mask R-CNN, arXiv:1703.06870) — the bilinear-sampled fixed-size crop that turns an arbitrary region (a proposal box (x1,y1,x2,y2)) of a feature map into a fixed PooledH x PooledW x Depth output, the pooling primitive a two-stage detector (Faster / Mask R-CNN) feeds to its box and mask heads. Unlike RoIPool it does no coordinate quantization: each output bin is the average of sampling_ratio² points sampled at the bin sub-centers by bilinear interpolation of the feature map (the exact sampler and per-corner gradient scatter of TNNetDeformableConv, summed over sample points and divided by the per-bin sample count). Matches torchvision.ops.roi_align with aligned=True (pixel-center half-pixel offset) and spatial_scale; sampling_ratio ≤ 0 selects the adaptive ceil(roi_size/pooled_size) ratio. Forward pinned against a torchvision-equivalent oracle (` |
README 内容过长，已截取前 64 KB。完整内容请前往 GitHub 仓库查看。
评论