LLM Security Deep Dive: Mathematical Components, Limitations & Attack Vectors (Part 2)

Introduction

This article is a continuation of The Model: From a Security Perspective 1. To equip you in understanding AI systems from scratch, we are going to show you the mathematical techniques, methodologies and combinations creating these magical thinking machines.

To lay the foundation for discovering different AI attacking vectors and potential security vulnerabilities, we are going to also analyze the limitations and accuracies of these mathematical components on computer systems and how these accumulated errors lead to certain behaviors and potential vulnerabilities you can exploit ethically.

AI Mathematics: The Building Blocks

AI advancements have heavily relied on theoretical foundations provided by mathematics and statistics. AI systems are made up of many mathematical functions forming different neural network patterns and architectures.

Picture being in the world of MINECRAFT, a popular 3D sandbox video game where you have to survive the night, explore the challenging world, and build anything with blocks based on your imaginations.

Blocks can be broken, crafted, placed to reshape the landscape, or used to build fantastical creations. This is the same for different mathematical components. They appear useless, yet skillfully using and combining different components can lead to solving many problems.

Mathematics is the functional building block of all software systems behind the scene. Investing in studying these building blocks can make you reap endless life benefits in the AI field and other fields as well.

From a security perspective, knowing the characteristics, limitations, and weaknesses of many blocks (mathematical components) will enable you to invent creative ways and techniques on how to break and build systems.

To enable you to understand the mathematical components powering these AI models, we will break down each mathematical theory, its applications, functions, limitations, and vulnerabilities.

Note: All the mathematical components covered in this article series don't justify the fact that there aren't more advanced mathematical components used in AI at the time of reading this post. You may want to check for missing components, as they are not exhaustive. However, the listed components are critical and widely applied. Although there might be quantum alternatives at the time of reading this post, these are the classical components.

Linear Algebra: The Dynamic Neural Linker

This is a branch of mathematics concerned with vectors, matrices, tensors, and linear transformations. Most layers in a neural network are basically a sequential operation of linear algebra.

Linear algebra helps AI systems in:

Compact representation (representing millions of datasets efficiently)
Parallel computation (speeding up computation by applying matrix methods in GPUs and TPUs)
Mathematical predictability (enforcing stable, deterministic, and analyzable linear operations)
Feature interaction (linking model relationships among multiple features by matrix multiplication)
Basis for differentiation (making backpropagation possible)

Limitations & Disadvantages: Although linear algebra is essential, certain limitations occur that can pose a security risk. The application of linear algebra normally leads to linear assumptions in AI models, which can cause wrong answer predictions. Although non-linear activation functions such as ReLU (Rectified Linear Unit) are used, other factors, such as numerical instability, cause overflow and underflow in floating-point arithmetic, especially during gradient updates (updating values to minimize loss).

Also, there is a problem of high memory cost, which is common in AI models due to storing matrices. It is also hard to understand relationships of linear transformations in high dimensions, as too many linear parameters might make the model memorize data rather than generalize (the overfitting problem).

Security Risk: Leveraging forward propagation and gradient calculation, attackers can exploit the model by crafting adversarial inputs (tiny perturbations). Due to linear approximations, a small change in input can cause a large change in output. This attack is applicable in image classification.

Data poisoning is also a major way this model can be exploited. Since AI relies on linear algebra to compute weights, well-placed malicious entries can manipulate the system, especially in the training matrix, leading to skewed results.

Another interesting vulnerability from this limitation is model inversion and extraction. Understanding deep linear algebra properties can allow an attacker to infer private data from models. This trend has been seen in recent jailbreaking exercises as models tend to leak sensitive information if queried with enough and well-structured inputs. Attackers can also leverage floating-point inputs to trigger misclassifications due to rounding errors.

Linear Algebra & Calculus Neural Network

Calculus: The Dynamic Sensors

Calculus plays a major role in AI systems. It provides the foundation for:

Optimization (enables the model to learn via gradient-based optimization)
Continuous modeling (modeling of probabilities, activation functions, and distributions)
Sensitivity analysis (the degree of model sensitivity to outputs based on changes in inputs or parameters)

Limitations & Disadvantages: Considering that calculus is an essential powering component in AI model systems, the limitations of calculus in numerical computations lead to approximation errors. Also, dependence on differentiability which doesn't correlate with real-world non-smooth phenomena leads to some issues in gradient-based optimizations. These limitations open doors for possible security exploits.

Security Risk: From the analyzed limitations, we can see that carefully crafted input changes which are possibly imperceptible to humans can drastically alter outputs. This is very applicable in image generation.

Also, attackers can carry out derivative-based attacks on recommendation systems to reveal private data or user preferences. Another vector of exploits can be through leveraging numerical instabilities in systems to craft large or small inputs that destabilize system training by causing overflow or underflow in gradient computation.

All these methodologies lead to model poisoning as the gradient is steered in malicious directions, causing the model to converge to harmful local minima in the loss landscape.

Probabilistic Theory: The Predictor

This is one of the core pillars of AI models as it is very applicable in machine learning, decision making, and reasoning under uncertainty. This powerful mathematical component allows an AI model to reason with incomplete or noisy data.

This theory has strong applications in machine learning and inferences such as:

Bayesian Networks (representation of dependencies from variables using probability, e.g., speech recognition and Part of Speech tagging in natural language processing)
Gaussian Mixture Models (GMMs) (model data distributions for clustering or density estimation, usually applied in image segmentation and anomaly detection)
Uncertainty Models (e.g., self-driving cars use probabilistic models to estimate the likelihood of pedestrian trajectories)
Decision making and reinforcement learning (probabilistic models estimate expected rewards and uncertainty in reinforcement learning)
Natural Language Processing (NLP) (models leverage probabilistic theory to predict next word and classify segments, e.g., n-gram models or transformers incorporate probabilities over sequences)
Sensor fusion & robotics (usage of probabilistic filters such as Kalman filters and particle filters to estimate position and states from noisy sensor data)
Anomaly detection systems (fraud and intrusion detection networks)

Probabilistic theory plays a major role in generative AI models.

Generative AI models use probabilistic theory to generate new data by sampling from trained data distributions. This provides the foundation of creativity in AI systems such as Variational Autoencoders (VAEs) [encoding of data probabilistically and generating new ones] and Diffusion Models (models sampling from learned distributions to generate realistic images).

Limitations & Disadvantages: Although probabilistic theory is inevitable in modern AI systems, there are general problems of computational cost, risk of overfitting data, data dependency, and assumption sensitivity which may not hold in real-world cases, leading to hallucination in real-world AI systems.

Security Risk: From the mathematical deficiencies, the limitations of probabilistic theory applications in AI systems lead to different security attack vectors in modern AI systems such as:

Adversarial attacks (generating misclassified inputs the model accepts as correct input even when wrong)
Poisoning attacks (manipulation of trained data to alter the model results)
Model inversion (repeated brute-force of model to reveal private data)
Sampling exploits (exploiting predictable regions or regions of low-entropy to get controlled generated outputs)
Robustness issues (tricking the model to be overly sensitive or have high confidence for a type of data even when incorrect)
Randomness exploits due to low entropy in AI-generated contents and cryptography-adjacent systems

Statistics: The Sensory Integrator

Statistics is a major part of AI with major dominance in the field of data science. It involves major steps in data flow such as data collection, analysis, interpretation, inference, and finally prediction.

An AI system is made up of a combination of applied statistical concepts and logic powered by computational power.

The following are different statistical concepts and their applications in the AI field:

Probabilistic Theory: This is used to model uncertainty, random behaviors, and predictions in AI systems.
Regression Analysis: This is used to predict relationships between variables. It can be linear regression or logistic regression.
Bayesian Inference: This area of statistics is applied in calculating hypothesis probability and updating the model as more information arrives in the system. It is actually applied in reinforcement learning.
Hypothesis Testing: This is used in model validation to check significance of results.
Sampling and Estimation: This is applied in stochastic training (randomly shuffling dataset for training) by using mini-batch gradient descent (division of dataset into small batches to compute model error and update parameters) and Monte Carlo methods (depending on repeated randomness sampling to obtain numerical results).
Descriptive Statistics: This is applied in summarizing dataset characteristics which can involve mean, variance, correlations, standard deviations, etc.
Statistical Distributions: This statistical method is used in modeling noise, activation randomness, and uncertainty (Normal, Bernoulli, Poisson, etc.).

Applications of Statistics

Data Analysis and Model Processing: Statistics are used to detect outliers and biases before a model is trained, normalize or standardize data, and perform feature selection through correlation variance and information gain.
Machine Learning Algorithms: Most machine learning algorithms are statistical in nature. Linear Regression estimates parameters through least squares (minimize statistical error), Logistic Regression uses statistical sigmoid mapping for classification, Naive Bayes Classifier (an ML classification algorithm that predicts data point categories using probability) uses Bayes' theorem which is statistical in nature, Decision Trees (a choice and outcome mapper) operate based on information gain, and SVMs (Support Vector Machines), supervised ML algorithms for classification and regression tasks, rely on maximizing statistical margins.
Deep Learning: Although deep learning looks numerical, it's deeply statistical. Properties such as weight initialization use Gaussian or uniform distribution. Loss functions such as cross-entropy are based on statistical divergence, e.g., Kullback-Leibler divergence (measurement of how one probability distribution diverges from another reference distribution). Batch normalization computes statistical mean and variance per batch. Dropout uses statistical probabilities to prevent overfitting.
Probabilistic AI & Bayesian Networks: Statistical modeling influences models such as Hidden Markov Models (HMMs) [probabilistic model used to analyze unobservable sequential data state to observable outputs], Kalman Filters (an efficient recursive filter estimating the internal dynamic state of linear dynamic systems from a series of noisy measurements), and Bayesian Networks (a probabilistic graphical model representing sets of variables and their conditional dependencies in a one-way connection linking graph called directed acyclic graph, i.e., DAG).
Reinforcement Learning: Statistics are applied in RL to estimate expected rewards, model state-action probabilities, and are also applied in Monte Carlo (experiential learning) or Temporal-Difference learning methods.
Generative Models: In generative models, statistical methodologies such as Gaussian latent distribution are used in VAEs (Variational Autoencoders). Diffusion models depend on noise statistics while GANs (Generative Adversarial Networks) use adversarial statistical equilibrium between generators and discriminators.

In summary, statistics have played a major role in AI systems to enhance data efficiency, predictive power, error control, model validation, and mathematical interpretability. However, based on real-world cases, there are bound to be limitations.

Limitations & Disadvantages: Although statistical models are heavily used in AI systems, they face limitations based on assumptions of linearity and independence which aren't true in real-world cases. There is also a risk of data bias propagation which is usually amplified when the dataset is biased. Overfitting is also possible coupled with unexplainable gaps due to layered applications in deep models. Finally, there is sensitivity risk to outliers enabling the model to be easily distorted with few data points and difficulty in scaling.

Security Risk: From the disadvantages and limitations, it is very obvious that the system is likely susceptible to different attacks such as:

Model inversion attack (use output statistics to reverse engineer inputs, e.g., contrastive prompting techniques)
Membership inference attack (predicting existing data points in training set through observing statistical outputs)
Data poisoning (manipulating data points to shift/change model's behavior)
Adversarial attack (exploiting statistical distribution to cause misclassification of inputs)
Privacy leak attack (leveraging overfitted models to extract sensitive patterns)
Bias amplification attack (encoding social bias to model for prediction)
Model fingerprinting (analyzing model outputs to infer architecture and parameters)

Conclusion

Bravo! You have completed part 2 of the series. The top 4 mathematical components above—namely algebra, calculus, probabilistic theory, and statistics—are the foundational mathematical components.

Although these components are intertwined in different types of AI models, we are decoupling each component one by one to give you the solid foundation to reverse engineer neural networks in future series.

Prepare for more mathematical components demystified in the next series: The Model: From a Security Perspective 3.

Ready to Secure Your AI Systems?

Now that you understand both the cognitive foundations (Part 1) and mathematical vulnerabilities in AI models, you might be wondering: "How do I actually audit and secure my AI systems in practice?"

At Zealynx, we specialize in comprehensive AI security assessments that go beyond traditional smart contract audits. Our team applies the cognitive security framework and mathematical analysis you've just learned—to identify vulnerabilities in:

LLM Applications - Prompt injection, context manipulation, data extraction
AI Agent Systems - Multi-modal attacks, tool misuse, privilege escalation
ML Pipeline Security - Training data poisoning, model extraction, adversarial inputs
AI Infrastructure - API security, access controls, deployment vulnerabilities

What makes our AI audits different:

Deep understanding of cognitive attack vectors (like those covered in this series)
Mathematical analysis of model behaviors and failure modes (linear algebra exploits, gradient attacks, probabilistic weaknesses)
Practical remediation strategies tailored to your AI architecture
Ongoing security monitoring and threat intelligence

Learn more about our AI Security Services →

FAQ

What are adversarial attacks in AI systems?

Adversarial attacks are techniques where attackers craft tiny, imperceptible perturbations to input data that cause AI models to misclassify or produce incorrect outputs. These exploit the mathematical properties of neural networks, particularly linear algebra approximations, where small input changes can cause large output changes. Common in image classification, these attacks demonstrate fundamental vulnerabilities in how AI systems process information.

What is model poisoning and how does it work?

Model poisoning is a security attack where malicious data is injected into the training dataset to manipulate the AI model's behavior. By exploiting how linear algebra and statistics are used to compute weights, attackers can place well-crafted malicious entries in the training matrix that cause the model to produce skewed results or behave incorrectly on specific inputs while appearing normal otherwise.

What is gradient descent in AI and why is it vulnerable?

Gradient descent is an optimization algorithm that uses calculus to minimize a model's error by iteratively adjusting weights based on the gradient (derivative) of the loss function. It's vulnerable because numerical instabilities in floating-point arithmetic can cause overflow or underflow, and attackers can craft inputs that manipulate these gradients to steer the model toward harmful local minima or destabilize training.

What is model inversion and how can it expose private data?

Model inversion is an attack technique that exploits statistical and probabilistic properties to reverse-engineer private training data from model outputs. By systematically querying an AI model and analyzing output patterns through statistical inference, attackers can reconstruct sensitive information the model was trained on, even if the model never directly reveals that data.

What is overfitting in AI models and why is it a security risk?

Overfitting occurs when an AI model memorizes training data rather than learning to generalize patterns. This happens when too many parameters (from linear algebra) cause the model to fit noise in the data. From a security perspective, overfitted models are vulnerable to privacy leaks (revealing training data), membership inference attacks (detecting if specific data was used in training), and can be more easily manipulated by adversarial inputs.

How do mathematical limitations create AI security vulnerabilities?

Mathematical limitations in AI create security vulnerabilities through several mechanisms: floating-point arithmetic causes rounding errors that attackers can exploit; linear approximations allow small input changes to cause large output changes (adversarial attacks); numerical instabilities in gradient computations enable model poisoning; and statistical assumptions that don't hold in real-world scenarios create attack vectors like model inversion and sampling exploits.

What is backpropagation and how can it be exploited?

Backpropagation is the algorithm that uses calculus (chain rule) to calculate gradients and update neural network weights during training. It can be exploited through gradient-based attacks where adversaries leverage the gradient calculation process to craft adversarial inputs, extract model information, or poison the training process by manipulating how errors propagate backward through the network.

How does probability theory enable AI model attacks?

Probability theory enables several AI attacks: sampling exploits target predictable or low-entropy regions in generative models; model inversion uses probabilistic inference to extract training data; adversarial inputs exploit predictable probability distributions; and randomness weaknesses in probabilistic sampling can be leveraged to manipulate AI-generated outputs or bypass security measures in cryptographic applications.

LLM Security Deep Dive: Mathematical Components, Limitations & Attack Vectors (Part 2)

Introduction

AI Mathematics: The Building Blocks