Random Signals and Stochastic Processes

Every signal in the previous twenty-eight chapters has been a known function. We wrote \(x(t) = e^{-2t}u(t)\), or \(x[n] = \cos(0.2\pi n)\), transformed it, filtered it, and read off the answer. That habit conceals an assumption so complete it is easy to miss: to compute a Fourier transform you must already know the signal for all time, past and future.

Consider what that assumption means for the systems of Chapter 28. A radio link exists to convey a message the receiver does not already know — if it did, there would be no reason to transmit. Shannon's observation is exactly this: information and unpredictability are the same thing seen from two sides, and a perfectly predictable signal conveys nothing. The moment a signal is worth transmitting, it cannot be written down in advance.

The second failure is more concrete. Every resistor in a receiver generates a fluctuating voltage from the thermal agitation of its charge carriers. Every amplifier adds shot noise from the discreteness of charge. Neither can be written as a formula, and neither can be measured once and reused, because the next measurement gives a different waveform. Yet these signals decide whether a link works, so a theory that can say nothing about them is a theory that cannot design a receiver.

There is also a mathematical obstruction, and Chapter 2 anticipated it. A noise waveform runs forever with roughly constant strength, so its total energy \(\int|x(t)|^2dt\) diverges — it is a power signal, not an energy signal. Chapter 13 required absolute integrability, or at least finite energy, for the Fourier transform to exist. For a noise sample function no such integral converges, and \(X(j\omega)\) simply does not exist. Even if we forced one into existence, it would be different for every sample function and therefore useless as a description of the noise.

A random variable \(X\) is a rule assigning a number to each outcome of an experiment. It is described completely by its cumulative distribution \(F_X(x) = P\{X \le x\}\), or equivalently by the probability density \(f_X(x) = dF_X/dx\), whose integral over an interval gives the probability of landing in it and whose total integral is 1.

Three averages carry almost all the weight in what follows. The expectation is the probability-weighted mean; the mean square is the expectation of \(X^2\); and the variance measures spread about the mean.

The last identity is worth verifying once, because the whole of Section 29-6 rests on it: expanding \((X-\mu)^2 = X^2 - 2\mu X + \mu^2\) and taking expectations term by term, using \(E[X] = \mu\), gives \(E[X^2] - 2\mu^2 + \mu^2 = E[X^2] - \mu^2\). Expectation is linear — that single fact is what makes every derivation in this chapter possible, and it is the direct analogue of the superposition that made Chapters 6 to 8 work.

Two densities appear repeatedly. The uniform density is constant over an interval and models a phase of which nothing is known. The Gaussian density is the model for thermal noise, and its pre-eminence is not a matter of convenience: the central limit theorem says that the sum of many small independent contributions tends to a Gaussian whatever the individual distributions are, and thermal noise is precisely the sum of an enormous number of independent electron motions.

One property of the Gaussian will matter in Section 29-8: a Gaussian random variable driven through a linear operation stays Gaussian. Since an LTI system computes a convolution — a weighted sum of input values — Gaussian noise entering a filter emerges Gaussian, with only its mean and variance altered. No other common density has this stability, and it is the reason a receiver can be analysed with two numbers rather than a full distribution.

A random process \(X(t)\) assigns not a number but a whole function of time to each outcome. Perform the experiment — switch on one particular resistor, at one particular moment — and you obtain one waveform, called a sample function. Perform it again and you obtain a different one. The entire collection of possible waveforms, together with their probabilities, is the ensemble, and the ensemble is what the theory describes. The single waveform on your oscilloscope is one draw from it.

The picture below is worth holding onto, because it contains the two directions in which averages can be taken, and confusing them is the commonest error in this subject.

Freeze the time at \(t_1\). The value \(X(t_1)\) is not yet a number, because we have not said which sample function was drawn; it is a random variable, with a density of its own. Freeze at a second instant \(t_2\) and you get a second random variable, generally correlated with the first. A random process is therefore an indexed family of random variables, one for every instant, and a complete description would require the joint density of every finite collection of them — an object nobody can write down.

Engineering practice retreats to the first two moments, and finds that they are enough for everything linear.

Note carefully what the expectation in these definitions means. It is an average across the ensemble, at fixed time: a vertical average in the figure. It is not an average along a single waveform. Whether the two coincide is the question of Section 29-5, and the answer is not automatic.

A process whose statistics change with time is intractable. What is needed is the probabilistic version of the time-invariance that Chapter 5 imposed on systems: the assumption that the mechanism generating the signal does not care what time it is. A resistor at constant temperature satisfies it; a resistor being heated does not.

Strict-sense stationarity states the assumption in full: every joint density is unchanged by a shift of the time origin, so that \(X(t_1),\dots,X(t_n)\) and \(X(t_1+\tau),\dots,X(t_n+\tau)\) have the same joint density for every \(n\) and every \(\tau\). This is a very strong requirement and almost never verifiable in practice. The workable weakening asks only that the first two moments be shift-invariant.

Strict-sense stationarity implies wide-sense, since if every density is shift-invariant then so is every moment computed from it. The converse fails in general — a process can have constant mean and shift-invariant correlation while its third moment drifts — with one important exception: for a Gaussian process the mean and autocorrelation determine every joint density completely, so wide-sense stationarity implies strict-sense. Since thermal noise is Gaussian, the distinction rarely troubles a communications engineer.

Two examples fix the idea. The random-phase sinusoid \(X(t) = A\cos(\omega_0 t + \Theta)\) with \(\Theta\) uniform on \([0,2\pi]\) is WSS: Worked Example 1 shows that its mean is zero and its autocorrelation is \((A^2/2)\cos\omega_0\tau\), with no residual dependence on \(t\). Remove the random phase, however, and \(X(t) = A\cos\omega_0 t\) is a deterministic signal whose "ensemble" has one member; its mean \(A\cos\omega_0 t\) varies with time, and it is not stationary. The random phase is what makes the process indifferent to the choice of time origin, which is a fair description of what stationarity means physically.

There is a practical gap in everything above. The definitions call for averages across an ensemble, but a laboratory has one resistor and one oscilloscope, and the ensemble is a mathematical fiction. What is actually measurable is a time average along one long record.

A process is ergodic when these time averages equal the corresponding ensemble averages, for almost every sample function. Ergodicity is the licence to substitute one long observation for the ensemble, and without it the theory would be unusable — every result in Sections 29-6 to 29-9 is stated in ensemble terms and applied to measured records.

The standard counterexample makes the gap vivid. Let \(X(t) = A\) for all time, with \(A\) drawn once, uniformly on \([-1,1]\). The process is stationary: its mean is \(E[A] = 0\) at every instant and \(R_x(\tau) = E[A^2] = 1/3\) for every \(\tau\). But each sample function is a constant, so its time average is \(A\) itself — which is 0 only for the one realisation that happens to have drawn \(A=0\). Averaging along a record here reveals the realisation, not the process. The physical reading is that the randomness was exhausted at the start and never renewed; ergodicity requires a process that keeps producing fresh randomness as time passes, and a single frozen draw does not.

Practically, ergodicity is why measured spectra look the way they do. Chapter 26 gave the FFT of a finite record; applied to a noise waveform, that transform is itself random and jumps about wildly from one record to the next. Averaging many such records, or equivalently smoothing in frequency, converges to the power spectral density of the next section — and it does so because the process is assumed ergodic.

For a WSS process the autocorrelation \(R_x(\tau) = E[X(t)X(t+\tau)]\) carries an unexpected amount of information, and four properties let it be read almost by inspection.

Setting \(\tau = 0\) gives \(R_x(0) = E[X^2(t)]\), the mean-square value — which for a voltage across one ohm is the average power. That single evaluation is the reason the autocorrelation, rather than any other second moment, is the object of interest.

Symmetry follows from stationarity. Since the statistics do not depend on the time origin, \(R_x(-\tau) = E[X(t)X(t-\tau)] = E[X(t')X(t'+\tau)] = R_x(\tau)\) after writing \(t' = t-\tau\). The autocorrelation of a real WSS process is an even function, always.

The bound \(|R_x(\tau)| \le R_x(0)\) comes from a device worth remembering, because the same trick proves the Cauchy–Schwarz inequality. The quantity \(E\big[(X(t) \pm X(t+\tau))^2\big]\) is an expectation of a square and therefore cannot be negative. Expanding it gives \(2R_x(0) \pm 2R_x(\tau) \ge 0\), which is the bound with both signs at once.

These four facts turn the graph of \(R_x(\tau)\) into a diagnostic chart. The peak at the origin is the total average power. The level the curve settles to far from the origin is the squared mean, the DC power. The width of the central peak is the correlation time — how long the process remembers, or how far ahead it can be usefully predicted. Any periodic component in the process survives in the autocorrelation as an undecaying periodic component of the same period, because a sinusoid stays correlated with itself forever; this is how a periodic signal buried in noise is detected.

The two panels above are one Fourier pair, and the inverse relation between their widths is the same time–bandwidth reciprocity that Chapter 14 established for deterministic signals: a process with a short memory has a wide spectrum, and a process with a long memory is spectrally narrow. That observation is the content of the next section, stated precisely.

We still want a frequency-domain description, and Section 29-1 established that the transform of a sample function does not exist. The way out is to transform something that does exist. The autocorrelation of a WSS process is a deterministic, even, bounded function which decays for most processes of interest, so it has a perfectly ordinary Fourier transform. Defining the power spectral density as that transform is the content of the Wiener–Khinchin theorem.

The name is justified by setting \(\tau = 0\) in the inverse transform. The left side becomes \(R_x(0)\), the average power; the right side becomes the total area under \(S_x(\omega)/2\pi\). So \(S_x(\omega)\) really does distribute the average power over frequency, and the power in any band is obtained by integrating over that band alone. This is Parseval's relation of Chapter 14 in its correct form for power signals.

Three properties of \(S_x(\omega)\) follow immediately and should be checked on every answer. It is real and even, because \(R_x(\tau)\) is real and even. It is non-negative for every \(\omega\) — a negative power density in some band would be meaningless, and the formal proof runs through the filtering result of the next section, by passing the process through a narrow bandpass filter and observing that the output power cannot be negative. And its area, divided by \(2\pi\), is the total power. A candidate spectral density that is odd, or dips below zero, is wrong.

An alternative and equally valid definition builds the density from finite records rather than from the autocorrelation. Truncate a sample function to \([-T,T]\), call the result \(x_T(t)\), which now has finite energy and therefore a transform \(X_T(j\omega)\), and form the average of \(|X_T|^2/2T\) over the ensemble as \(T\) grows.

The expectation is essential. Without it the quantity \(|X_T|^2/2T\) — the periodogram — does not converge as \(T\) grows: its fluctuation stays as large as its mean no matter how long the record. Lengthening the record buys finer frequency resolution but no reduction in variance whatever, which is a genuinely counter-intuitive fact and the reason every practical spectrum estimator averages many short periodograms instead of computing one long one. Chapter 26's FFT supplies the periodograms; the averaging supplies the convergence.

The discrete-time statements are identical with sums in place of integrals. For a WSS sequence, \(R_x[m] = E\{x[n]x[n+m]\}\) and \(S_x(e^{j\Omega}) = \sum_m R_x[m]e^{-j\Omega m}\), which — like every discrete-time spectrum in Chapter 16 — is periodic in \(\Omega\) with period \(2\pi\). The average power is \(R_x[0] = (1/2\pi)\int_{-\pi}^{\pi}S_x(e^{j\Omega})\,d\Omega\).

Five pairs cover most of what is met in practice, and they are worth knowing by sight in the way the transform pairs of Chapter 13 are.

This is the section the chapter exists for. A WSS process enters a stable LTI system with impulse response \(h(t)\); what comes out? Chapter 7 gave \(y(t) = \int h(\alpha)x(t-\alpha)\,d\alpha\) for each individual sample function, and since convolution is linear, expectations pass straight through it.

Take the mean first. Expectation is linear and \(h\) is deterministic, so it commutes with the integral:

The output mean is the input mean times the DC gain — which is exactly what it must be, since a constant is a signal at zero frequency and the system responds to it with \(H(0)\).

Now the autocorrelation. Write \(y(t)\) and \(y(t+\tau)\) as two convolution integrals, multiply, and take the expectation inside both:

The step that makes this work is the second equality: because the input is WSS, the inner expectation depends only on the difference of the two time arguments, \((t+\tau-\beta)-(t-\alpha) = \tau+\alpha-\beta\), and the absolute time \(t\) has vanished. The output is therefore WSS as well — a conclusion worth stating on its own, since it means the machinery is closed under filtering.

The double integral is a pair of convolutions in disguise, and it is far easier to read after transforming. Take the Fourier transform with respect to \(\tau\), and exchange the order of integration:

using the shifting property on the inner integral. The remaining double integral now factors completely: the \(\alpha\) part is \(\int h(\alpha)e^{j\omega\alpha}d\alpha = H(-j\omega) = H^*(j\omega)\) for real \(h\), and the \(\beta\) part is \(\int h(\beta)e^{-j\omega\beta}d\beta = H(j\omega)\). Their product is \(|H(j\omega)|^2\).

One more correlation is worth defining, because it turns the result into a measurement technique. The cross-correlation between output and input is \(R_{yx}(\tau) = E[Y(t+\tau)X(t)]\), and running the same argument with only one convolution to expand gives \(R_{yx}(\tau) = h(\tau) * R_x(\tau)\).

The idealisation that makes the previous section practical is white noise: a process whose power spectral density is flat, \(S_x(\omega) = N_0/2\) for all \(\omega\). The name borrows from white light, which contains all visible frequencies in equal measure. Transforming back gives an autocorrelation that is an impulse.

Read the second statement literally and its two faces appear at once. Any two distinct instants are uncorrelated, however close together — the process has no memory at all. And \(R_x(0)\) is infinite, so white noise has infinite average power and cannot exist. It is a fiction, and a legitimate one, for the same reason the impulse of Chapter 3 was: what is actually required is that the density be flat over the band the system responds to, and every real system has a finite bandwidth. Thermal noise is flat to roughly \(10^{12}\) Hz, which is white enough for any circuit anyone is building.

The physics fixes the constant. A resistor at absolute temperature \(T\) delivers an available noise power spectral density of \(kT\) watts per hertz, one-sided, with \(k = 1.38\times10^{-23}\) J/K; equivalently \(N_0 = kT\) and the two-sided density is \(N_0/2 = kT/2\). At room temperature \(N_0 \approx 4\times10^{-21}\) W/Hz, a number worth remembering, since it sets the floor beneath which no receiver can be designed.

Because white noise is flat, the output power of a filter driven by it depends only on the area under \(|H|^2\). That observation is packaged as the noise-equivalent bandwidth: the width of the ideal brick-wall filter, of the same peak gain, that would pass the same noise power.

With noise power in hand, the figure of merit that decides whether a link works is the signal-to-noise ratio, \(\text{SNR} = P_{\text{signal}}/P_{\text{noise}}\), usually quoted in decibels as \(10\log_{10}\) of that ratio. Every design decision in Chapter 28 can now be re-read as an SNR decision: AM's carrier wastes power that would otherwise raise the SNR; FM's wide bandwidth admits more noise but the discriminator's nonlinearity converts the extra bandwidth into an SNR gain proportional to \(\beta^2\); PCM's regenerators refuse to accumulate noise at all, which is why a digital link either works or does not, with very little in between.

One final result closes the circle with Chapter 15. Suppose a known pulse \(s(t)\) of duration \(T\) arrives buried in white noise, and we may choose any filter we like to maximise the output signal-to-noise ratio at the sampling instant. The answer is not a lowpass filter, and it does not depend on the noise beyond its density: the optimum is the matched filter \(h(t) = s(T-t)\), the time-reverse of the pulse itself.

Problems 1 to 3 test stationarity and the reading of autocorrelations; 4 to 6 push signals through filters; 7 and 8 are design calculations of the kind a receiver requires. Where a spectral density is asked for, check that your answer is real, even and non-negative before going on.