On notation for Hamiltonians and variations
Here we summarize the notational clashes that we encountered in our derivation. There is no need to read this section since we discussed these during the derivation. Here we only state it for convenient reference.
Hamiltonian
In our derivation we had to form the augmented Lagrangian. There were two ways to do it. We could write it as L^\text{aug}(\bm x,\bm u,\boldsymbol \lambda, t) = L(\bm x,\bm u,t)+ \boldsymbol \lambda^\top (t)\left[\mathbf f(\bm x,\bm u,\mathbf t)-\dot {\bm x}(t)\right], but we could also easily formulate it as \tilde L^\text{aug}(\bm x,\bm u,\boldsymbol \lambda, t) = L(\bm x,\bm u,t)+ \tilde{\boldsymbol \lambda}^\top (t)\left[ \dot {\bm x}(t) - \mathbf f(\bm x,\bm u,\mathbf t)\right].
Both are clearly correct. Indeed, although the intermediate steps differ, the final results (Riccati equation, optimal state feedback gains) are identical.
The former choice, the one that we used in our derivation, L^\text{aug}(\bm x,\bm u,\boldsymbol \lambda, t) = H(\bm x,\bm u,\hat{\boldsymbol \lambda}, t) - \boldsymbol \lambda^\top (t) \dot {\bm x}(t), where \boxed{ H(\bm x,\bm u,\boldsymbol \lambda,t) = L(\bm x,\bm u, t)+\boldsymbol \lambda^\top (t) \mathbf f(\bm x,\bm u,t).}
The latter choice, which can also be encountered in the literature, enables us to write the augmented Lagrangian using a Hamiltonian as \tilde L^\text{aug}(\bm x,\bm u,\boldsymbol \lambda, t) = \tilde {\boldsymbol \lambda}^\top (t) \dot {\bm x}(t) - \tilde H(t,\bm x,\bm u,\tilde {\boldsymbol\lambda}) where the Hamiltonian is defined as \boxed{ \tilde H(\bm x,\bm u,\tilde {\boldsymbol \lambda}, t) = \tilde{\boldsymbol \lambda}^\top (t) \mathbf f(\bm x,\bm u,t) - L(\bm x,\bm u, t).}
Whether one or the other, the canonical equations are identical. It is only that the second-order sufficiency conditions show maximization of the Hamiltonian in one case and minimization in the other. This can be concluded by observing that H(\bm x,\bm u,\boldsymbol \lambda, t) = -\tilde H(\bm x,\bm u,\tilde{\boldsymbol \lambda}, t).
Also \tilde{\boldsymbol \lambda} = -\boldsymbol \lambda.
While I confess I hesitated whether or not this complication due to two different definitions of the Hamiltonian is worth mentioning in our introductory text, the reality is that both conventions can be encountered in the literature and one should be aware of it. Most standard books rarely warn the reader about it. One of a few discussions of this frequent source of notational confusion is in [1, Sec. 3.4.4].
Variation
Upon consulting numerous textbooks and monographs, it appears that the authors are far from the accord regarding the definition of variation (within the context of calculus of variations). Two main definitions appear:
The one that we followed in this lecture defines the variation as an extension of the concept of a differential. That is, a variation \delta J of a (cost) functional is the first-order approximation to the increment \Delta J in the (cost) functional J. This we discussed in quite some detail in the text.
The other one defines variation as the derivative of the (cost) functional with respect to the real (perturbation) parameter. In our text, it is the \frac{\mathrm{d}J}{\mathrm{d}\alpha} (for fixed y(x) and \eta(x)) that would be called a variation and labelled \delta J. The increment in the (cost) functional would be then be approximated by \delta J\;\alpha.
Both definitions are often encountered in the literature, but we prefer the former because the definitions of variation of a functional \delta J and variation of a function \delta y(x) are consistent. It is also the convention followed by the popular reference [2], while the latter definition is followed by [1].