دانلود کتاب راهنمای مبانی ریاضی شبکه‌های عصبی

1 Matrix algebra and numerical methods 1
1.1 Recommended literature………………………………………………………………………….. 1
1.2 Norms……………………………………………………………………………………………………… 1
1.2.1 Applications of Norms in Numerical Mathematics…………………………. 2
1.3 Systems of Linear Equations and Direct Methods……………………………………. 3
1.4 Gaussian Elimination without Row Replacement……………………………………. 4
1.5 Gaussian Elimination with Partial Pivoting……………………………………………. 6
1.5.1 Advantages of Gaussian Elimination with Partial Pivoting…………….. 8
1.5.2 Full Pivoting…………………………………………………………………………………. 9
1.6 Matrix Decompositions…………………………………………………………………………….. 9
1.6.1 LU Decomposition………………………………………………………………………… 9
1.6.2 QR Decomposition…………………………………………………………………………….12
1.6.3 Cholesky Decomposition…………………………………………………………………….18
1.6.4 Grout decomposition………………………………………………………………………….20
1.6.5 Singular Value Decomposition (SVD) ………………………………………………..22
1.7 Least Squares Method…………………………………………………………………………………27
2 Interpolation 32
2.1 Recommended literature……………………………………………………………………………32
2.2 Introduction to Interpolation………………………………………………………………………32
2.3 Polynomial Interpolation………………………………………………………………………………32
2.3.1 Lagrange Interpolation………………………………………………………………………33
2.3.2 Newton Interpolation …………………………………………………………………….36
2.4 Error of Polynomial Interpolation…………………………………………………………………38
2.4.1 Selecting Points to Minimize Error…………………………………………………….38
2.5 Definitions of Linear, Bilinear, and Trilinear Interpolation……………………………39
2.6 Radial Basis Functions Interpolation……………………………………………………………..41
2.7 Spline Interpolation …………………………………………………………………………………….42
2.7.1 Linear Splines…………………………………………………………………………………….42
2.7.2 Cubic Splines…………………………………………………………………………………….43
2.8 Examples…………………………………………………………………………………………………….45
3 Iterations and stability review 51
3.1 Recommended literature………………………………………………………………………………51
3.2 Difference equations……………………………………………………………………………………51
3.3 Equilibrium Definition…………………………………………………………………………………58
3.4 Stability Definitions ……………………………………………………………………………………58
3.5 Criterion for Asymptotic Stability ………………………………………………………………61
3.6 Destabilization through Bifurcations…………………………………………………………..62
3.7 Definition of a Cobweb Plot…………………………………………………………………………63
3.8
Linear Homogeneous Difference Equations with Constant Coefficients … 65
3.9 Leslie Model………………………………………………………………………………………………….67
3.10 Markov Chains…………………………………………………………………………………………..68
3.11 Power Method ………………………………………………………………………………………..72
3.12 Autonomous Systems of Nonlinear Difference Equations……………………………73
3.12.1 Equilibrium Points……………………………………………………………………………73
3.12.2 Linearization of Nonlinear Difference Systems……………………………………73
IV
3.12.3 Criteria for Stability of Equilibria…………………………………………………..74
4 Fixed Point Iterative Methods 75
4.1 Recommended literature……………………………………………………………………………….75
4.2
Fixed Point Iterative Methods in the Context of Difference Equations … 75
4.2.1 Problem Definition…………………………………………………………………………….75
4.2.2 Convergence and Stability………………………………………………………………….76
4.3 Newton method…………………………………………………………………………………………80
4.3.1 Motivation ……………………………………………………………………………………….80
4.3.2 Newton Method for a Single Nonlinear Equation………………………………81
4.3.3 Newton Method for Systems of Nonlinear Equations………………………..82
4.3.4 Numerical Continuation Using One Parameter ………………………………..87
4.3.5 Gauss-Newton Method………………………………………………………………………89
4.4 Methods derived from Newton method…………………………………………………………91
4.4.1 Secant Method………………………………………………………………………………….91
4.4.2 Regula Falsi Method………………………………………………………………………….92
4.4.3 Quasi-Newton Methods for Multiple Variables……………………………………93
4.5 Rate of Convergence…………………………………………………………………………………..94
4.5.1 Proof of Quadratic Convergence of Newton Method ……………………….100
4.5.2 Basins of Attraction arid criteria of convergence……………………………….101
4.5.3 Big O and Little o Notation…………………………………………………………….103
4.5.4 Convergence Acceleration and Steffensen Method……………………………..104
5 Iterative solving of systems of linear equations 109
5.1 Recommended literature……………………………………………………………………………..109
5.2 Introduction……………………………………………………………………………………………109
5.3 Jacobi and Gauss-Seidel Methods……………………………………………………………….Ill
5.3.1 Jacobi Method……………………………………………………………………………….Ill
5.3.2 Gauss-Seidel Method……………………………………………………………………….112
5.4 Criteria of convergence………………………………………………………………………………..113
5.4.1 Diagonal Dominance Theorem ………………………………………………………..114
5.4.2 Positive Definiteness Theorem…………………………………………………………..115
5.5 Relaxation Method……………………………………………………………………………………..115
5.5.1 Optimal Selection of Relaxation Parameter…………………………………….115
5.6 Two-Grid method………………………………………………………………………………………116
5.6.1 Defect correction principle………………………………………………………………..116
5.6.2 Two-Grid Method …………………………………………………………………………..117
5.7 Examples………………………………………………………………………………………………..118
6 Optimization 130
6.1 Recommended literature……………………………………………………………………………..130
6.2 Optimization in TZ……………………………………………………………………………………..130
6.2.1 Analytic Method……………………………………………………………………………..130
6.2.2 Simple Division Method…………………………………………………………………..131
6.2.3 Bisection Method……………………………………………………………………………..131
6.2.4 Golden Ratio Method……………………………………………………………………..132
6.2.5 Computational complexity of comparative methods………………………….132
6.2.6 Fibonacci Method…………………………………………………………………………..134
6.2.7 Golden Ratio vs. Fibonacci……………………………………………………………..135
6.3 Quadratic Interpolation Method ……………………………………………………………….136
V
6.3.1 Examples………………………………………………………………………………………..138
6.3.2 Solution: ………………………………………………………………………………………..139
6.4 Optimization in TZn……………………………………………………………………………………..141
6.5 Nelder-Mead Method………………………………………………………………………………….141
6.5.1 Algorithm…………………………………………………………………………………………141
6.6 Gradient Descent Method………………………………………………………………………….143
6.6.1 Algorithm………………………………………………………………………………….143
6.7 Newton Method……………………………………………………………………………………….144
6.7.1 Local extremes in R”………………………………………………………………………..144
6.7.2 Algorithm…………………………………………………………………………………………144
6.8 Quasi-Newton Methods …………………………………………………………………………….145
6.8.1 Broyden-Fletcher-Goldfarb-Shanno (BFGS) Algorithm……………………..145
6.9 Conjugate Gradient Method……………………………………………………………………….147
6.9.1 Algorithm………………………………………………………………………………………..147
6.9.2 Computation of ak and /3k ……………………………………………………………..147
6.10 Examples……………………………………………………………………………………………..148
7 Numerical integration 151
7.1 Recommended literature………………………………………………………………………..151
7.2 Quadrature Formulae ……………………………………………………………………………….151
7.3 Newton-Cotes formulae…………………………………………………………………………….151
7.3.1 Composite Rules……………………………………………………………………………..153
7.4 Gaussian Quadrature Formulae………………………………………………………………….154
7.5 Gauss quadrature formulae ……………………………………………………………………….155
7.5.1 Gauss-Legendre Integration…………………………………………………………….155
7.5.2 Gauss-Chebyshev Integration………………………………………………………….158
7.6 Monte Carlo integration…………………………………………………………………………….159
8 Numerical differentiation and solving differential equations 162
8.1 Recommended literature…………………………………………………………………………….162
8.2 Numerical Computation of Derivative ………………………………………………………162
8.2.1 Derivative Approximation Using Interpolation Polynomial………………162
8.2.2 Derivative Approximation Using Taylor Series…………………………………163
8.3 Numerical Solution of Ordinary Differential Equations………………………………164
8.3.1 Explicit Euler Method……………………………………………………………………..164
8.3.2 Implicit Euler Method……………………………………………………………………..164
8.3.3 Predictor-Corrector Methods………………………………………………………….164
8.3.4 Solving ODEs Using Runge-Kutta Methods…………………………………….165
8.4 Second-Order Differential Equations with Boundary Conditions………………….166
8.5 Numerical Solution of Partial Differential Equations…………………………………166
8.5.1 Discretization of the Spatial and Temporal Domains………………………167
8.5.2 Finite Difference Method ………………………………………………………………..167
8.5.3 Explicit and Implicit Schemes………………………………………………………….168
8.5.4 Boundary and Initial Conditions…………………………………………………….169
8.6 Examples………………………………………………………………………………………..170