# Advanced Robotics

**Advanced Robotics: Analytical Dynamics, Optimal Control, and Inverse Optimal Control.** These documents develop legged and floating robotics from the bottom up, starting with a study of the fundamental building blocks of control design, analytical dynamics, and continuing through to floating-based instantaneous control, optimal control, and imitation learning through inverse optimal control. I originally wrote them for the first half of a course I taught on Advanced Robotics at the University of Stuttgart, summer semester 2014.

**Linear Algebra and Robot Modeling. **Linear algebra is fundamental to robot modeling, control, and optimization. This document reviews some of the basic kinematic equations and uses them to motivate an SVD-centric geometric perspective on linear algebra. This perspective illuminates the underlying structure and behavior of linear maps and simplifies analysis, especially for reduced rank matrices. We later review some properties of multidimensional quadratic forms that are central to optimal control. Most of this material should already be familiar to the reader; this document explores some specifics and offers a potentially unique intuition oriented perspective.

**Some Notes on Advanced Calculus.** This document briefly discusses the pragmatic side of Advanced Calculus. We review notational conventions, some tricks of the trade, and some intuition behind the Inverse Function Theorem

**Analytical Dynamics and Contact Analysis.** Robots in a vacuum are useless--- we care about robots insomuch as they can be used to interact with the environment. Understanding contact forces, their analysis within a robotic system, and how they can be controlled is, therefore, critical. This document discusses why we need the Analytical Dynamics of Lagrange, Hamilton, and Gauss for robot analysis and builds to a discussion of contact forces in a dynamical system. We'll see that Lagrangian mechanics provides a coordinate independent approach to classical mechanics, liberating Newton's second law f = ma from its inertial reference frames and making it applicable to the generalized coordinates of robotic joint angles. We'll also see how both the conservation of momentum and the conservation of energy are fundamental consequences of the equations themselves, and how contact constraints in the environment can be modeled using constrained optimization techniques. This discussion culminates in an analysis of contact constraints for floating-based rigid body robotic systems and derive the constrained equations of motion that form the backbone for controller used in today's most advanced robotics systems.

**Controlling Floating-Based Robots**. Floating-based legged robots are some of the most impressive and fascinating robots in existence. But they're also some of the most difficult to control. This document presents and analyzes the constrained equations of motion for a floating-based robot in contact with the environment. We discuss the issues that arise when we detach the robot from the world and make it mobile, and we examine the linear structure connecting dynamic quantities such as accelerations, torques, and contact forces to one another. After getting our hands dirty with objective design for control systems, exploring in detail the specific problem of controlling a robot's center of mass and some of the difficulties that arise in that context with designing objectives alone, we present the state-of-the-art building block for controller design, Quadratic Programming, the centerpiece of most of today's most advanced control systems. Almost all advanced torque controlled robots today are built on these tools and dynamic principles.

**Nonlinear Optimal Control: Reductions to Newton Optimization.** This document explores nonlinear optimal control from the perspective of generic nonlinear optimization. We discuss first why optimization is important to control and see one case, in particular, where optimizing future performance leads to a simple control algorithm that differs in a very interesting and intuitive way from an analogous PD control rule that we might have written down directly. Building from that example, we then develop some nonlinear optimal control algorithms and principles more concretely and show that the natural approach of iteratively solving second-order approximations to the problem parallels a more general methodology of Newton's method for unconstrained nonlinear optimization. In other words, classic algorithms such as DDP and iLQR may be understood simply as variants on Newton's method that exploit the underlying structure of the discrete time optimal control problem. This observation makes available a broad set of tools for solving these problem such as line search techniques, trust region methods, and nonlinear methods from constrained optimization. To this end we discuss the basic principles behind Newton's method for general nonlinear optimization and derive an inequality constrained variant of the Augmented Lagrangian algorithm and discuss their applicability to optimal control.

**Stochasticity in Nonlinear Optimal Control.** Deterministic dynamics is a severe abstraction. We can never correctly predict precisely what's going to happen when we apply a given set of torques to a complex high dimensional robotic system, especially when the robot has significant inertia and is in contact with the unstructured environment. Assuming some level of stochasticity help us reason about these uncertainties and even choose actions to explicitly reduce uncertainty. This document studies stochasticity within nonlinear optimal control. We start by exploring to to properly model stochastic dynamics in a way that leads to action choices explicitly accounting for uncertainty. We derive a very general framework for reasoning about stochasticity over a long horizon in optimal control that allows us to leverage both general nonlinear optimization tools and common distributional approximations such as Expectation Propagation to ensure that the distribution over where the robot might be doesn't grow arbitrarily complex as it propagates along the trajectory. We then discuss how various modeling tools (entropy costs and Gaussian expectations) interaction with the optimization to produce explicit uncertainty reducing behaviors to better achieve desired goals.

**Inverse Optimal Control: What do we optimize?** How do we design the behavior of an autonomous system? Optimal control says, give us a cost function and we'll give you an optimal policy. Can't get any better than that, right? So we go to work designing a cost function. But for all our efforts to stay principled, we've all been there, in the lab, late at night, tuning parameters, weighing cost components, designing cost features, and tweaking, tweaking, tweaking to get the darned thing to do what we want. Optimal control methods are optimizers, but in reality, we don't really know what we should be optimizing. This document addresses the question of how we can, instead, leverage our strong intuition of how we'd like the robot to behave to learn a cost function that embodies demonstrations of that behavior. So defines Inverse Optimal Control (IOC). Rather than computing an optimal policy given a cost function, IOC uncovers the cost function that best explains demonstrated optimal behavior. This problem is strongly related to a broad class of very large scale multiclass classification problems (Structured Prediction) in machine learning, and we start by deriving and discussing that connection in detail. We then derive a simple gradient-based method for learning that that solves IOC. The algorithm consists of an outer loop optimization that repeatedly solves the optimal control problem under successively improving hypotheses and subsequently corrects errors the perceived behavior. The algorithm effectively automates the manual guess and check process that plagues the design of optimal controllers.