Numerical Bayesian Methods Applied to Signal Processing

1 Introduction.- 2 Probabilistic Inference in Signal Processing.- 2.1 Introduction.- 2.2 The likelihood function.- 2.2.1 Maximum likelihood.- 2.3 Bayesian data analysis.- 2.4 Prior probabilities.- 2.4.1 Flat priors.- 2.4.2 Smoothness priors.- 2.4.3 Convenience priors.- 2.5 The removal of nuisance parameters.- 2.6 Model selection using Bayesian evidence.- 2.6.1 Ockham's razor.- 2.7 The general linear model.- 2.8 Interpretations of the general linear model.- 2.8.1 Features.- 2.8.2 Orthogonalization.- 2.9 Example of marginalization.- 2.9.1 Results.- 2.10 Example of model selection.- 2.10.1 Closed form expression for evidence.- 2.10.2 Determining the order of a polynomial.- 2.10.3 Determining the order of an AR process.- 2.11 Concluding remarks.- 3 Numerical Bayesian Inference.- 3.1 The normal approximation.- 3.1.1 Effect of number of data on the likelihood function.- 3.1.2 Taylor approximation.- 3.1.3 Reparameterization.- 3.1.4 Jacobian of transformation.- 3.1.5 Normal approximation to evidence.- 3.1.6 Normal approximation to the marginal density.- 3.1.7 The delta method.- 3.2 Optimization.- 3.2.1 Local algorithms.- 3.2.2 Global algorithms.- 3.2.3 Concluding remarks.- 3.3 Integration.- 3.4 Numerical quadrature.- 3.4.1 Multiple integrals.- 3.5 Asymptotic approximations.- 3.5.1 The saddlepoint approximation and Edgeworth series.- 3.5.2 The Laplace approximation.- 3.5.3 Moments and expectations.- 3.5.4 Marginalization.- 3.6 The Monte Carlo method.- 3.7 The generation of random variates.- 3.7.1 Uniform variates.- 3.7.2 Non-uniform variates.- 3.7.3 Transformation of variables.- 3.7.4 The rejection method.- 3.7.5 Other methods.- 3.8 Evidence using importance sampling.- 3.8.1 Choice of sampling density.- 3.8.2 Orthogonalization using noise colouring.- 3.9 Marginal densities.- 3.9.1 Histograms.- 3.9.2 Jointly distributed variates.- 3.9.3 The dummy variable method.- 3.9.4 Marginalization using jointly distributed variates.- 3.10 Opportunities for variance reduction.- 3.10.1 Quasi-random sequences.- 3.10.2 Antithetic variates.- 3.10.3 Control variates.- 3.10.4 Stratified sampling.- 3.11 Summary.- 4 Markov Chain Monte Carlo Methods.- 4.1 Introduction.- 4.2 Background on Markov chains.- 4.3 The canonical distribution.- 4.3.1 Energy, temperature and probability.- 4.3.2 Random walks.- 4.3.3 Free energy and model selection.- 4.4 The Gibbs sampler.- 4.4.1 Description.- 4.4.2 Discussion.- 4.4.3 Convergence.- 4.5 The Metropolis-Hastings algorithm.- 4.5.1 The general algorithm.- 4.5.2 Convergence.- 4.5.3 Choosing the proposal density.- 4.5.4 Relationship between Gibbs and Metropolis.- 4.6 Dynamical sampling methods.- 4.6.1 Derivation.- 4.6.2 Hamiltonian dynamics.- 4.6.3 Stochastic transitions.- 4.6.4 Simulating the dynamics.- 4.6.5 Hybrid Monte Carlo.- 4.6.6 Convergence to canonical distribution.- 4.7 Implementation of simulated annealing.- 4.7.1 Annealing schedules.- 4.7.2 Annealing with Markov chains.- 4.8 Other issues.- 4.8.1 Assessing convergence of Markov chains.- 4.8.2 Determining the variance of estimates.- 4.9 Free energy estimation.- 4.9.1 Thermodynamic integration.- 4.9.2 Other methods.- 4.10 Summary.- 5 Retrospective Changepoint Detection.- 5.1 Introduction.- 5.2 The simple Bayesian step detector.- 5.2.1 Derivation of the step detector.- 5.2.2 Application of the step detector.- 5.3 The detection of changepoints using the general linear model.- 5.3.1 The general piecewise linear model.- 5.3.2 Simple step detector in generalized matrix form.- 5.3.3 Changepoint detection in AR models.- 5.3.4 Application of AR changepoint detector.- 5.4 Recursive Bayesian estimation.- 5.4.1 Update of position.- 5.4.2 Update given more data.- 5.5 Detection of multiple changepoints.- 5.6 Implementation details.- 5.6.1 Sampling changepoint space.- 5.6.2 Sampling linear parameter space.- 5.6.3 Sampling noise parameter space.- 5.7 Multiple changepoint results.- 5.7.1 Synthetic step data.- 5.7.2 Well log data.- 5.8 Concluding Remarks.- 6 Restoration of Missing Samples in Digital Audio Signals.- 6.1 Introduction.- 6.2 Model formulation.- 6.2.1 The likelihood and the excitation energy.- 6.2.2 Maximum likelihood.- 6.3 The EM algorithm.- 6.3.1 Expectation.- 6.3.2 Maximization.- 6.4 Gibbs sampling.- 6.4.1 Description.- 6.4.2 Derivation of conditional densities.- 6.4.3 Conditional density for the missing data.- 6.4.4 Conditional density for the autoregressive parameters.- 6.4.5 Conditional density for the standard deviation.- 6.5 Implementation issues.- 6.5.1 Estimating AR parameters.- 6.5.2 Implementing the ML algorithm.- 6.5.3 Implementing the EM algorithm.- 6.5.4 Implementation of Gibbs sampler.- 6.6 Relationship between the three restoration methods.- 6.6.1 ML vs Gibbs.- 6.6.2 Gibbs vs EM.- 6.6.3 EM vs ML.- 6.7 Simulations.- 6.7.1 Autoregressive model with poles near unit circle.- 6.7.2 Autoregressive model with poles near origin.- 6.7.3 Sine wave.- 6.7.4 Evolution of sample interpolants.- 6.7.5 Hairy sine wave.- 6.7.6 Real data: Tuba.- 6.7.7 Real data: Sinead O'Connor.- 6.8 Discussion.- 6.8.1 The temperature of an interpolant.- 6.8.2 Data augmentation.- 6.9 Concluding remarks.- 6.9.1 Typical interpolants.- 6.9.2 Computation.- 6.9.3 Modelling issues.- 7 Integration in Bayesian Data Analysis.- 7.1 Polynomial data.- 7.1.1 Polynomial data.- 7.1.2 Sampling the joint density.- 7.1.3 Approximate evidence.- 7.1.4 Approximate marginal densities.- 7.1.5 Conclusion.- 7.2 Decay problem.- 7.2.1 The Lanczos problem.- 7.2.2 Biomedical data.- 7.2.3 Concluding remarks.- 7.3 General model selection.- 7.3.1 Model selection in an impulsive noise environment.- 7.3.2 Model selection in a Gaussian noise environment.- 7.4 Summary.- 8 Conclusion.- 8.1 A review of the work.- 8.2 Further work.- A The General Linear Model.- A.1 Integrating out model amplitudes.- A.1.1 Least squares.- A.1.2 Orthogonalization.- A.2 Integrating out the standard deviation.- A.3 Marginal density for a linear coefficient.- A.4 Marginal density for standard deviation.- A.5 Conditional density for a linear coefficient.- A.6 Conditional density for standard deviation.- B Sampling from a Multivariate Gaussian Density.- C Hybrid Monte Carlo Derivations.- C.1 Full Gaussian likelihood.- C.2 Student-t distribution.- C.3 Remark.- D EM Algorithm Derivations.- D.l Expectation.- D.2 Maximization.- E Issues in Sampling Based Approaches to Integration.- E.1 Marginalizing using the conditional density.- E.2 Approximating the conditional density.- E.3 Gibbs sampling from the joint density.- E.4 Reverse importance sampling.- F Detailed Balance.- F.1 Detailed balance in the Gibbs sampler.- F.2 Detailed balance in the Metropolis Hastings algorithm..- F.3 Detailed balance in the Hybrid Monte Carlo algorithm..- F.4 Remarks.- References.