You Underestimate the Power of RANDOMNESS

Central to these potential advancements and lucrative ventures is the notion of RANDOMNESS. Contrary to popular belief, randomness does not equate to chaos or absolute unpredictability. In reality, randomness adheres to specific mathematical principles. We must reevaluate our perception of randomness, beginning with its mathematical foundation: the moment generating function (MGF), closely tied to the exponential function.

The MGF of a random variable X serves as a potent tool, encapsulating the essence of a probability distribution. It is defined as the expected value of the exponential function applied to X:

While this may sound technical, the elegance lies in its applications. By expanding the MGF via a Taylor series, we unlock an infinite sequence of moments that elucidate everything from the average outcome (mean) to variability (variance) and beyond:

The initial moments are particularly significant: Mx?(0) is the first derivative of the MGF evaluated at t=0, yielding the mean ?. The second derivative MX?(0) Mx???(0) provides the second moment, related to the variance ?² upon deducting the square of the mean. However, the MGF extends further. For instance, the third derivative Mx???(0) corresponds to the third moment, indicating skewness, which reveals the distribution's asymmetry. The fourth derivative Mx?(0) yields the fourth moment, associated with kurtosis, illustrating the "tailedness" or extremity of the distribution's tails. This progression continues, providing an infinite array of moments, each enriching our understanding of the distribution and revealing new statistical descriptors.

By combining the straightforward MGF, the Taylor expansion, and a hidden theorem (which we will soon unveil), we can derive this elegant formula for calculating any moment within our probabilistic distribution:

Here, the first moment represents the MEAN: E[Xi?]=0, and the second moment, the variance:

The third moment, the skewness of our distribution, follows suit:

E[Xi³] = (1 — p)³ * p + (-p)³ * (1 — p) = p — 3p² + 2p³

What about the fourth moment? You might have anticipated this:

E[Xi?] = (1 — p)? * p + (-p)? * (1 — p) = p — 4p² + 6p³ — 3p?

These mathematical wonders allow us, through comparison, to grasp the entirety of a narrative by merely scanning a few pivotal lines. By utilizing the MGF and Taylor expansion, one can delve into the core of a probability distribution—regardless of its shape—unearthing critical attributes such as the mean, variance, and more, all without needing the complete formula. It’s a brilliant shortcut that facilitates understanding the essence of the distribution merely by examining its moments.

Indeed, you no longer need extensive data or an expansive data center for processing overwhelming AI—just your tech-savvy intellect and a compact laptop!

Upon grasping this powerful tool, you can adapt your collection of random variables typically framed within your stochastic processes, which are central to your use cases or business models. For instance, this could apply in scenarios involving combinatorial graphs (networking, transportation challenges, cybersecurity, robotics, smart grids, cloud computing), financial risk management in volatile markets, or quality assessment in industrial production—the applications are numerous.

Consider a case where we aim to compute the expected probability of an extreme deviation for our business, given a threshold (k). For instance, our business might involve real-time optimization of a smart grid, where a group of power generator nodes (dominant group) is connected to all distribution points serving millions during extreme seasonal weather conditions. The central dilemma is how to anticipate the expected extreme deviation in current demand during an extremely cold winter or hot summer and adjust power production accordingly. This scenario encapsulates an NP problem (the combinatorial connections between power producers and distribution nodes) nested within another NP problem (the multiplication of numerous exponential functions) involving combinatorial, graph, and set theories—and it becomes even more complex over time!

Don't fret; in the next article, you will discover how to tackle this problem using bound combinatorics and random variables.

Forget the concept of BIG DATA, and steer clear of current Artificial Intelligence—relying on either will not yield the precise responses you require, even with a large and costly data center operating at peak capacity. But it is this remarkable tool that will turn your day around.

Here’s the approach: Abstract your problem, whether it's a power grid or highly volatile financial markets that don’t conform to the well-known Gaussian or Normal distribution due to frequent extreme events (crashes, price spikes, or energy consumption). The subsequent step is to apply the moment method mentioned earlier, alongside a theorem that provides bounds on the probability of such large deviations.

Advancing into Concentration Inequalities

Concentration inequalities serve as mathematical instruments that describe how a random variable (or a summation of random variables) deviates from a central value, typically its expected value. These inequalities provide bounds on the probabilities that the random variable strays significantly from this central value. The term “concentration” conveys the concept that the values of the random variable cluster around its mean or expected value, and these inequalities quantify the likelihood of substantial deviations from the mean.

1. Concentration Around the Mean:
   • Concentration inequalities assist in understanding how closely a random variable is clustered around its mean or expected value.
   • They offer upper bounds on the probabilities of deviations, indicating how likely (or unlikely) it is for the variable to take values far from the mean.
2. Quantifying Deviations:
   • These inequalities are employed to quantify the probability of significant deviations, outlining the rate at which the probability diminishes as the deviation from the mean expands.
3. Ensuring High Probability of Closeness:
   • Concentration inequalities guarantee that, with high probability, the random variable will be near its expected value. This is vital in probabilistic analysis, statistical inference, and machine learning, where the stability of empirical averages is often relied upon.

Notable Concentration Inequalities

1. Markov’s Inequality:
   • Applicable to any non-negative random variable X and a > 0.
2. Chebyshev’s Inequality:
   • Suitable for any random variable X with mean ? and variance ?² and for any k > 0.
3. Chernoff Bounds:
   • For a summation of independent Bernoulli random variables ?i = ?Xi?, where Xi ~ Bernoulli(p), the Chernoff bound provides exponential limits on the tail probabilities for right and left deviations.
4. Alon-Spencer Theorem:
   • For independent random variables X1, X2,…, Xt with specific distributions, the Alon-Spencer theorem delivers bounds on the probability that their summation deviates significantly below the mean.

We can utilize either the Alon-Spencer Theorem for negative extreme deviations or the Chernoff Theorem for both negative and positive deviations. In this scenario, we opt for the simplicity of the Alon-Spencer Theorem to compute the probability that our energy production falls below a designated k-threshold (the same logic applies if you're interested in a market where extreme prices can result in losses surpassing a given threshold).

The term a signifies the magnitude of the deviation from the mean (expected value). It represents the distance you are considering beneath the mean for which you wish to constrain the probability.

The term 2tp in the denominator correlates with the variance of the summation of the random variables. The Alon-Spencer theorem offers an exponential limit on the probability of large deviations for summations of random variables like X.

To apply this straightforward Alon-Spencer Theorem, we need the expected value or mean of our random variables to be zero. How do we achieve that? It's simple—by selecting appropriate probability values for our two outcomes. For instance, do our distributed power stations have the right number of connections to the generator power stations (yes or no), or in financial trading, can we anticipate our losses exceeding expectations given the market conditions (yes or no)? Moreover, given two outcomes, we can be more precise and utilize a binomial distribution as follows:

Definition of ?j?

?j? is a random variable that can take two values: 1?p and ?p

Probabilities of ?j?:

The expected value of ?j?: Now you can comprehend why we have selected those probabilities to render the expected value of ?j? zero.

Choosing 1?p and ?p for ?j? ensures that the random variable centers around zero, making it ideal for concentration inequalities such as the Alon-Spencer theorem.

This configuration balances the positive and negative contributions in such a way that the overall expected deviation equals zero, which is crucial for deriving precise bounds on the probability of large deviations.

In the next article here on Medium, you will witness how all these foundational concepts unfold to create a real-time adaptive solution for predicting the probability of extreme changes and enhancing your business profitability—from smart grid energy usage to navigating highly volatile markets. See you then!