Ja na ni V en ka ta su br am an ian Ta rg ete d ex pl or at io n an d ro bu st du al co nt ro l f or li ne ar d yn am ica l s ys tem s Janani Venkatasubramanian Targeted exploration and robust dual control for linear dynamical systems Designing controllers for uncertain dynamical systems requires accurate models to satisfactorily regulate system behaviour. Obtaining such models creates a fundamental trade-off between exploring the system to reduce model uncertainty while controlling the system, or exploiting the system, to achieve a desired performance, which constitutes the central challenge of dual control. In this thesis, this problem is addressed by developing a computationally tractable sequential dual control strategy for uncertain discrete-time linear time-invariant systems. A framework is presented that features a targeted exploration phase followed by a robust control phase, resulting in a robust dual control approach with a priori guarantees on excitation during exploration and closed- loop performance after exploration. Crucially, the excitation during the exploration phase is directly determined by the desired performance objective, thereby explicitly capturing the dual effect of performance improvement through the process of exploration. The presented dual control approach leverages data-dependent identification bounds based on the disturbances affecting the system for the design of targeted exploration, and robust gain-scheduling control. Targeted exploration and robust dual control for linear dynamical systems Von der Fakultät Konstruktions-, Produktions- und Fahrzeugtechnik der Universität Stuttgart zur Erlangung der Würde eines Doktor-Ingenieurs (Dr.-Ing.) genehmigte Abhandlung Vorgelegt von Janani Venkatasubramanian aus Chennai, Indien Hauptberichter: Prof. Dr.-Ing. Dr. h.c. Frank Allgöwer Mitberichter: Prof. Mark Cannon, DPhil Prof. Håkan Hjalmarsson, PhD Tag der mündlichen Prüfung: 10.10.2025 Institut für Systemtheorie und Regelungstechnik Universität Stuttgart 2026 Acknowledgements The completion of this dissertation marks an important milestone in my academic journey, and was made possible by the blessings, guidance, support, effort, and en- couragement of many people. First and foremost, Iwould like to expressmy sincere gratitude tomyPhDadvisor, Prof. Frank Allgöwer, for granting me the freedom to pursue my research interests and for his trust in my abilities. I am deeply grateful to him for his invaluable ad- vice and for creating a stimulating research environment at the Institute for Systems Theory and Automatic Control (IST), which provided me with the opportunity to interact with so many extraordinary researchers. I would like to thank Prof. Mark Cannon (University of Oxford) and Prof. Håkan Hjalmarsson (KTHRoyal Institute of Technology) for their interest inmywork, their valuable comments, and for serving on my doctoral examination committee. I am especially grateful to Prof. Mark Cannon for hostingme in Oxford for threemonths, for his generous hospitality, our collaboration, his valuable insights and his exem- plary approach to research and mentorship, all of which have shaped not only this dissertation but also my development as a researcher. I was fortunate to conduct research alongside exceptional colleagues and experts. I would like to thank Julian Berberich and Johannes Köhler (Imperial College Lon- don) for our collaboration, which began with my first project at the IST. I’m grate- ful for the valuable research discussions with them, and the outcomes of our work have been instrumental in shaping the results presented in this dissertation. Fur- thermore, I’m particularly grateful to Johannes Köhler for his long-term collabora- tion, and calm and steady guidance. I have truly learned a great deal from him, not only about conducting research rigorously but also about approaching complex problems thoughtfully and creatively. My time at the IST was very productive, owing to the stimulating research en- vironment and the support of my colleagues. I am grateful to them for the many iii research discussions during group meetings and seminars, impromptu conversa- tions, and their own inspiring research. Their feedback on my written work and rehearsal talks has been invaluable in significantly shaping both the development of this dissertation and my growth as a researcher. I am particularly grateful to Ju- lian Berberich, Felix Brändle, Nicolas Chatzikiriakos, Maximilian Degner, Johannes Köhler, Bowen Song, Robin Strässer, Dominik Tschemernjak and Yifan Xie for proof- reading this dissertation. I am grateful to the IST secretariat for their immense sup- port with administrative matters. I would also like to thank the IMPRS-IS coordina- tion team for their continuous support throughout my PhD. Moving to Germany for a PhDwas not easy, and adjusting to life in a new country posed its own set of challenges, but the support and generosity ofmy colleagues and friends in Stuttgart and abroadmademyPhDexperience truly rewarding. Tomy IST colleagues, thank you for your camaraderie, the amazing hikes, the swims in lakes, the cave exploration adventure andmany other outdoor activities that I would have never ventured into otherwise. To Lucrezia Manieri and Yana Lishkova, the fierce and wonderfully talented women I met in Oxford, thank you for making my time in Oxford memorable, and for inspiring me with your passion and determination. To Antje Jensch and Viviane Klingel, thank you for your friendship, the board game nights, and the conversations over several cups of tea. To Antje Jensch and Yifan Xie, thank you for being the best office mates one could ask for. To the WISR team, thank you for inspiring me and showing me that we can make a real difference in this world. To Tanmayee Narendra, who started and finished her PhD around the same time I did, thank you for your friendship, motivation and support throughout this journey. To Varun Sridhar, Shambhuraj Sawant, andHarish N. R., thank you for sharing this PhD journeywithme, for your emotional support, and for all the amaz- ing food. To my Stuttgart girls, particularly Jovitha Serrao, Karuna Soraganvi, Stefy Varghese and Nicci Sequeira, thank you for the joy and laughter, for celebrating the small things in life, and for being my home away from home. To Kanika Kharbanda and Aishwarya Karthikeyan, thank you for being my biggest cheerleaders despite the distance, for checking in to support me, and for always rooting for me. I am deeply grateful to Prof. H. N. Shankar and Prof. R. Muralishankar (both formerly at CMRIT, Bengaluru) for introducing me to the path of research and for their perpetual guidance. Were it not for them, I would never have reached this milestone. It has been an honour and a privilege to be their student. iv Finally and most importantly, I would like to thank my family for believing in me and for their unwavering support in my pursuits away from home. I am especially grateful tomy amma, Vijaya LakshmiRamakrishnan, and appa, Venkatasubramanian Krishnan, for standing by me in light and dark, and for continuously inspiring me to better myself. I am grateful to my paatis (grandmothers) for their strength and resolve, and for motivating me to pursue this path, one in which they would have excelled had they been given the opportunity in their time. I am deeply grateful to my husband, Aditya Srinivasan Tirunilai, for his patience, care, understanding, and his humour, which lightened even the hardest days. This journey has been smooth and fulfilling because of my family, who have been my constant source of strength. Bochum, January 2026 Janani Venkatasubramanian v Table of Contents Notation ix Abstract xiii Deutsche Kurzzusammenfassung xvii 1. Introduction 1 1.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2. Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3. Contributions and outline of the thesis . . . . . . . . . . . . . . . . . . 9 2. Preliminaries 15 2.1. Data-dependent uncertainty quantification . . . . . . . . . . . . . . . 15 2.2. Frequency domain information using spectral lines . . . . . . . . . . 24 2.3. Nominal and robust performance analysis . . . . . . . . . . . . . . . . 26 3. Targeted exploration for linear systems 33 3.1. Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.2. Asymptotic targeted exploration for systems with stochastic distur- bances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.3. Approximate targeted exploration for systems with energy-bounded disturbances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.4. Asymptotic targeted exploration for systems with energy-bounded disturbances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.5. Non-asymptotic targeted exploration for systems with stochastic dis- turbances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 3.6. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 vii Table of Contents 4. Robust dual control 87 4.1. Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.2. Robust gain-scheduling design . . . . . . . . . . . . . . . . . . . . . . 89 4.3. Dual control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.4. Numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.6. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5. Conclusions 111 5.1. Summary of contributions . . . . . . . . . . . . . . . . . . . . . . . . . 111 5.2. Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 A. Technical proofs 119 A.1. Deferred proofs from Chapter 2 . . . . . . . . . . . . . . . . . . . . . . 119 A.2. Deferred proofs from Chapter 3 . . . . . . . . . . . . . . . . . . . . . . 120 A.3. Deferred proofs from Chapter 4 . . . . . . . . . . . . . . . . . . . . . . 139 Bibliography 141 Publications of the author . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 viii Notation In what follows, we list the main symbols and acronyms used in this thesis. Addi- tional notation is defined in the corresponding sections. Abbreviations and acronyms cf. confer (compare) e.g. exempli gratia (for example) i.e. id est (that is) i.i.d. independent and identically distributed s.t. such that DFT discrete Fourier transform DP dynamic programming LMI linear matrix inequality LPV linear parameter-varying LQR linear quadratic regulator LTI linear time-invariant MAP maximum a priori MPC model predictive control SDP semidefinite program Symbols ≈ approximately equal := is defined as ∈ belongs to → tends to 7→ maps to ⊆ non-strict subset ix Table of Contents ∀ for all ∝ proportional to ⇐⇒ if and only if, or equivalent to max maximum min minimum sup supremum, the least upper bound inf infimum, the greatest lower bound lim limit log natural logarithm exp exponential cos cosine Sets and spaces {·} unordered set N set of integers N0 set of non-negative integers N>0 set of positive integers Rn space of n-dimensional vectors with real entries Rm×n space of matrices of dimensionm× nwith real entries Cn space of n-dimensional vectors with complex entries Cm×n space of matrices of dimensionm× nwith complex entries ℓn2 signal space ℓ2 := { x : N0 → Rn ∣∣√∑∞ k=0 x ⊤ k xk < ∞ } ℓn2e extended ℓ2 space ℓ2e := { x : N0 → Rn ∣∣ √∑N k=0 x ⊤ k xk < ∞, ∀N ∈ N0 } Sequences {xk}Nk=1 sequence of vectors xk ∈ Rn, N ∈ N>0 {x(ejωi)}Ni=1 discrete Fourier transform of the sequence {xk}Nk=1, N ∈ N>0, with x(ejωi) = ∑N−1 k=0 xke −j2πkωi where ωi ∈ ΩN := {0, 1/N, . . . , (N − 1)/N} x Table of Contents Random variables and probability distributions E[x] expected value of a random vector x ∈ Rn P[E] probability of an event E occurring χ2 n(δ) critical value of the Chi-squared distribution with n degrees of freedom and probability δ x ∼ N (µ,Σ) a Gaussian random vector x ∈ Rn with mean µ ∈ Rn and covariance Σ ∈ Rn×n, Σ ⪰ 0 x ∼ subG(µ,Σ) a sub-Gaussian random vector x ∈ Rn with mean µ ∈ Rn and variance proxy Σ ∈ Rn×n, Σ ⪰ 0 Vectors, matrices, and norms j imaginary unit, i.e., j2 = −1 1n vector of dimension nwith all entries equal to 1 In identity matrix of dimension n× n A⊤ transpose of a matrix A ∈ Rm×n AH conjugate transpose of a matrix A ∈ Cm×n A ≻ 0 (A ⪰ 0) matrix A ∈ Rn×n, or A ∈ Cn×n, is positive definite (positive semidefinite), i.e., A = A⊤ and x⊤Ax > 0 (x⊤Ax ≥ 0) for all x ∈ Rn with x ̸= 0 A ≺ 0 (A ⪯ 0) matrix A ∈ Rn×n, or A ∈ Cn×n, is negative definite (negative semidefinite), i.e., A = A⊤ and x⊤Ax < 0 (x⊤Ax ≤ 0) for all x ∈ Rn with x ̸= 0 A⊗B Kronecker product of matrices A ∈ Rm1×n1 and B ∈ Rm2×n2 [A,B] horizontal concatenation of matrices A ∈ Cm×n1 and B ∈ Cm×n2 , i.e, [A,B] = [A B] ∈ Cm×(n1+n2) |x| component-wise absolute value of the vector x ∈ Rn ⟨x, y⟩ Euclidean inner product of vectors x ∈ Rn and y ∈ Rn, i.e., ⟨x, y⟩ = x⊤y ∥x∥ Euclidean norm of x ∈ Rn, ∥x∥ = √ x⊤x ∥x∥P weighted Euclidean norm of x ∈ Rn and a positive definite matrix P = P⊤ ∈ Rn×n , ∥x∥P = √ x⊤Px xi Table of Contents ∥A∥ largest singular value of a matrix A ∈ Cm×n ∥A∥M given a matrixM ⪰ 0, ∥A∥M = ∥M1/2A∥where M1/2 is the symmetric square root matrix ofM vec(·) the operator vec(·) converts a matrix A ∈ Rm×n into a vector x = vec(A) ∈ Rmn by vertically stacking the columns of A diag(·) the operator diag(A1, . . . , An) creates a block diagonal matrix by aligning the matrices A1, . . . , An, n ∈ N>0, along the diagonal xii Abstract Control design of uncertain dynamical systems requires accurate models to satis- factorily regulate system behaviour. Obtaining such models poses a fundamental challenge of actively exploring the system to reduce uncertainty in the model while controlling the system, or exploiting the system, to achieve a desired performance. This challenge is the focus of dual control, which seeks to balance the conflicting ob- jectives of exploration and exploitation for uncertain systems. Typically, dual control methods employ heuristic approximations to decouple these two objectives, which can lead to insufficient excitation during exploration and a lack of formal perfor- mance guarantees. In this thesis, we address this problem by developing a compu- tationally tractable sequential dual control strategy for uncertain discrete-time lin- ear time-invariant (LTI) systems. Our approach features a targeted exploration phase, which systematically reduces uncertainty in the model, followed by the implemen- tation of a robust controller. We present a framework to simultaneously design tar- geted exploration and robust control, resulting in a robust dual control approach with a priori guarantees on excitation during exploration and closed-loop perfor- mance after exploration. Crucially, our approach ensures that the excitation during the exploration phase is directly determined by the desired performance objective, thereby explicitly capturing the dual effect of performance improvement through the process of exploration. The presented dual control approach leverages data- dependent identification bounds based on the disturbances affecting the system for the design of targeted exploration, and robust gain-scheduling control. To this end, we first present the targeted exploration framework and subsequently integrate it with robust gain-scheduling control to formulate the robust dual control approach. xiii Table of Contents Targeted exploration We present a novel framework for designing targeted exploration strategies tailored to specific disturbancemodelswithmulti-sine exploration inputs. We leverage data- dependent uncertainty bounds on the uncertain parameters, based on the distur- bances affecting the system, to derive sufficient conditions on the exploration data that guarantee a desired lower bound on the excitation of the exploration inputs. This lower bound, in turn, ensures a desired accuracy in the parameter estimates obtained through exploration. Building on this, we apply the theory of spectral lines to express sufficient conditions directly in terms of the exploration inputs. Fur- thermore, we robustly account for parametric uncertainty, the effect of disturbances, and spectral transient errors to formulate linear matrix inequalities for semidefinite program-based (SDP-based) targeted exploration strategies. In particular, we first address the setting of classical stochastic disturbances, i.e., independent, identically distributed, and Gaussian with zero mean and known covariance. The designed robust targeted exploration strategy yields a priori asymptotic guarantees on the accuracy of the estimated parameters. Subsequently, we address the case of energy- bounded disturbances by designing an initial computationally tractable finite-time exploration strategy using common approximations. Building on this, we present a robust targeted exploration strategy for systems subject to energy-bounded dis- turbances with asymptotic guarantees on parameter accuracy. Lastly, we derive a finite-time exploration strategy for systems subject to sub-Gaussian disturbances that ensures a desired accuracy in the parameters with high probability. The prac- tical benefits and applicability of the presented targeted exploration strategies are demonstrated with different numerical examples. Robust dual control We present a novel framework for designing a sequential robust dual control strat- egy for uncertain LTI systems. In order to tailor the exploration strategy to control performance objectives and encapsulate the dual effect, we jointly design targeted exploration and the robust controller. This design depends on the future model es- timate, which will be obtained through exploration, and its associated uncertainty bound. The changes in the model estimate through the process of exploration are explicitly accounted for by using tools from linear parameter-varying (LPV) sys- xiv Table of Contents tems and robust gain scheduling. In particular, the desired performance objective directly shapes the uncertainty bound on the future estimate and, consequently, the level and direction of excitation required during exploration. This ensures that the information gathered during exploration is sufficient for robust control. Overall, we present an SDP-based robust dual control approach, which integrates targeted ex- ploration and a gain-scheduling controller, that ensures a desired performance after exploration while minimizing exploration energy. We provide a detailed theoreti- cal analysis of the presented approach and demonstrate its effectiveness through a numerical example. xv Deutsche Kurzzusammenfassung Die Reglerauslegung für unsicherer dynamischer Systeme erfordert genaue Mod- elle, um das Systemverhalten zufriedenstellend zu regeln. Die Erstellung solcher Modelle stellt eine grundlegende Herausforderung dar, da das System aktiv explo- riert werden muss, um die Unsicherheit im Modell zu reduzieren. Gleichzeitig soll das System geregelt und damit das, in Form des Modells, vorliegende Wissen über das Systemverhalten ausgenutzt werden, um eine gewünschte Performance zu er- reichen. Diese widersprüchlichen Ziele in Einklang zu bringen, ist eine zentrale Problemstellung im Feld der dualen Regelung. Typischerweise verwenden duale Regelungsmethoden heuristische Näherungen, um diese beiden Ziele voneinander zu entkoppeln, was zu einer unzureichenden Anregung während der Exploration und einem Mangel an formalen Performancegarantien führen kann. In dieser Ar- beit wird dieses Problem gelöst indem eine sequenzielle Regelungsstrategie für un- sichere zeitdiskrete lineare zeitinvariante Systeme entwickelt wird. Unser Ansatz umfasst eine Phase der gezielten Exploration, in der die Unsicherheit im Modell systematisch reduziert wird. Darauf folgt die Implementierung eines robusten Re- glers. Dabeiwerdendie Strategien zur gezielten Exploration und robustenRegelung gemeinsam entworfen, was zu einem robusten dualen Regelungsansatz mit a priori Garantien für die Anregung während der Exploration und der daraus resultieren- den Performance im geschlossenen Regelkreis führt. Entscheidend ist, dass unser Ansatz sicherstellt, dass dieAnregungwährendder Explorationsphase direkt durch das gewünschte Regelziel bestimmtwird, wodurch der duale Effekt der Performance- verbesserung durch die Exploration explizit erfasst wird. Hierfür verwenden wir datenabhängige Schranken für den Identifikationsfehler basierend auf den Störun- gen. Zu diesem Zweck stellen wir zunächst ein Framework für die gezielte Ex- ploration vor und integrieren dieses anschließend in eine robuste Gain-Scheduling- Regelung, um einen robusten dualen Regelungsansatz zu formulieren. xvii Table of Contents Gezielte Exploration Wir stellen ein neuartiges Framework für die Entwicklung gezielter Multi-Sinus- Explorationsstrategien vor, welche auf verschiedene Charakterisirung der Störun- gen zugeschnitten sind. Aus den auf das System einwirkenden Störungen resul- tieren datenabhängige Unsicherheitsschranken für die Parameter, welche verwen- det werden um hinreichende Bedingungen für die Exploration abzuleiten, die ein gewünschtes Maß an Exploration garantieren. Diese Exploration wiederum gewährleistet eine gewünschte Genauigkeit der durch die Exploration gewonnenen Parameterschätzungen. Darauf aufbauend wenden wir die Theorie der Spek- trallinien an, um diese hinreichenden Bedingungen direkt in Bezug auf die Ex- plorationsinputs auszudrücken. Darüber hinaus berücksichtigen wir robust die parametrische Unsicherheit, die Auswirkungen von Störungen und spektrale tran- siente Fehler und nutzen lineare Matrixungleichungen um in Form von SDPs gezielte Explorationsstrategien zu formulieren. Insbesondere befassen wir uns zunächst mit klassischen Gauss-verteilten stochastischen Störungen. Die entwor- fene gezielte Explorationsstrategie liefert a priori asymptotische Garantien für die Genauigkeit der geschätzten Parameter. Anschließend befassen wir uns mit dem Fall energiebeschränkter Störungen, indem wir eine numerisch umsetzbare Finite- Time-Explorationsstrategie unter Verwendung gängiger Approximationen entwer- fen. Darauf aufbauend präsentieren wir eine robuste gezielte Explorationsstrategie für Systeme, die energiebeschränkten Störungen unterliegen, und präsentieren asymptotische Garantien für die Parametergenauigkeit. Zuletzt stellen wir eine Explorationsstrategie für Systeme vor, die subgaussischen Störungen unterliegen. Hierbei wird nach endlicher Zeit mit hoher Wahrscheinlichkeit eine gewünschte Genauigkeit der Parameter gewährleistet. Die praktischen Vorteile und die An- wendbarkeit der vorgestellten gezielten Explorationsstrategien werden anhand ver- schiedener numerischer Beispiele demonstriert. Robuste duale Regelung Wir stellen ein neuartiges Framework für die Entwicklung einer sequenziellen robusten Dual-Control-Strategie für unsichere LTI-Systeme vor. Um die Explo- rationsstrategie auf die Leistungsziele der Regelung abzustimmen und den dualen Effekt einzubeziehen, entwickeln wir die gezielte Exploration und den robusten xviii Table of Contents Regler gemeinsam. Dieser Entwurf hängt sowohl von der zukünftigen Mod- ellschätzung ab, welche durch die Exploration gewonnen wird, als auch von der damit verbundenen Unsicherheitsschranke. Die änderungen der Modellschätzung durch den Explorationsprozess werden mithilfe von Methoden aus linearen pa- rametervariablen (LPV) Systemen und robustem Gain-Scheduling explizit berück- sichtigt. Insbesondere bestimmt das gewünschte Performancesziel direkt die Un- sicherheitsgrenze der zukünftigen Schätzung und damit das Ausmaß und die Richtung der während der Exploration erforderlichen Anregung. Dadurch wird sichergestellt, dass die während der Exploration gesammelten Informationen für eine robuste Steuerung ausreichen. Insgesamt präsentierenwir einen SDP-basierten robustenDual-Control-Ansatz, der gezielte Explorationund einenGain-Scheduling- Controller integriert. Hierbei wird nach der Exploration die gewünschte Perfor- mance sichergestellt und gleichzeitig den Energieaufwand für die Exploration min- imiert. Wir liefern eine detaillierte theoretische Analyse des vorgestellten Ansatzes und demonstrieren seine Wirksamkeit anhand eines numerischen Beispiels. xix Chapter 1. Introduction 1.1. Motivation Modern engineered systems, from autonomous vehicles and drones to industrial robots and smart grids, frequently operate in environments that present significant uncertainty. In addition to uncertainty in the environment, these systems are often characterized by incomplete or inaccurate models of their own dynamics. While perfect knowledge of the environment and the system dynamics is rarely available, some level of knowledge is essential for designing reliable controllers that achieve desired performance objectives. Therefore, it is necessary to gather informative data to reduce uncertainty and learn about the environment and the dynamics up to a desired level of accuracy, within finite time or with limited resources. A widely used approach to design controllers in such settings relies on a math- ematical model of the system to predict its behaviour and determine appropriate control actions. In practice, this often involves a two-step process: first, a model is identified from input-output data through system identification techniques [95]; second, a controller is designed using the identified model by employing model- based techniques [143]. This indirect data-driven approach is typically grounded in the separation principle [40] and certainty equivalence (CE) control [83], where the controller is designed by assuming that the estimated model is the true system, effectively ignoring the residual uncertainty in the estimation. While this simplifica- tion greatly reduces design complexity in the case of small model errors, it can lead to performance degradation or instability in the case of large model errors. Robust control methods address this issue by utilizing a model estimate and its associated 1 Chapter 1. Introduction uncertainty bound to provide stability andperformance guarantees [158]. However, these guarantees may be conservative and performance may be poor, particularly when uncertainty bounds are conservative. System identification is the key enabler of model-based control strategies, either CE control strategies like the linear quadratic regulator (LQR) [85] and model pre- dictive control (MPC)[57], or robust control methods like robust MPC [17]. This involves constructing a state-space model of the underlying system based on input- output data. Although data for system identification is often inexpensive and abun- dant, high-quality informative data that reveals uncertainties in the system dynam- ics is significantly harder to obtain. In particular, when designing controllers with theoretical guarantees, the identified model must not only fit the data well but also be accurate enough to provide acceptable closed-loop performance. This notion has been captured by the field of identification for control [61] which focuses on estimat- ingmodels that are well-suited for their intended use in control design. This field of work stems from the idea that an estimated model is only an approximation of the true system dynamics, and the quality of the estimated model should depend on the model’s intended application. In this context, it is essential to not only estimate a model but also to quantify the model’s uncertainty in a way that is relevant for control design. However, determining rigorous uncertainty bounds is inherently challenging as they depend on the level of excitation, the presence of disturbances, the richness of data, and the identification method. Conservative bounds, which usually arise from poorly designed data collection, may lead to controllers with poor performance. Conversely, underestimating uncertainty can cause instability. This trade-off highlights the need for targeted data acquisition strategies and led to the concept of goal-oriented identification, where identification is viewed as a design problem tailored to control objectives. In this regard, optimal experiment design and targeted exploration [64, 69, 113] play a central role by strategically obtaining in- formative data from an experiment that is used to derive a model of the system. These methods seek to choose experimental conditions, such as inputs and the du- ration of the experiment, so that the resulting model and its associated uncertainty description enable robust control design with desired guarantees. However, much of the results in literature on optimal experiment design, in general, lack robustness and guaranteed bounds on excitation. While classical approaches separate model estimation and controller design, the 2 1.1. Motivation field of simultaneous identification and control seeks to unify these tasks by designing inputs that not only regulate the system but also actively gather informative data that reduces uncertainty about the system dynamics. Research interest in this field was established by the dual control paradigm introduced by A. A. Feldbaum in the 1960s [44, 45, 46, 47]. This pioneering work recognized that control inputs to an uncertain system have a ‘probing’ effect to learn the uncertainty in the system, and a ‘directing’ effect to control the dynamical system. However, these two effects are naturally conflicting, drawing attention to the trade-off between ‘exploration’ (i.e., learning and reducing system uncertainty) and ‘exploitation’ (i.e., controlling the system to achieve optimal performance), which is also the subject of contemporary literature on reinforcement learning [116]. Dual control relies on stochastic dynamic programming (DP) [16] which is, however, computationally intractable and there- fore, not applicable to real-world problems. Hence, designing controllers which capture the ‘dual effect’ have relied on either approximations of stochastic DP, or other approaches to solve the problem of tractability. In summary, existing optimal experiment design and targeted exploration meth- ods in literature, while performing well numerically, lack guarantees on excitation and robustness. These methods, when combined with robust control design, fail to provide theoretical performance guarantees. This limitation motivates the work in this thesis. We present a tractable dual control approach that captures the essen- tial exploration-exploitation trade-off by designing targeted exploration and robust control inputs simultaneously. For linear systems with either stochastic or bounded disturbances, we derive a priori guarantees on the excitation of the exploration in- puts. The presented results include asymptotic guarantees that are valid as the ex- ploration time tends to infinity, and non-asymptotic guarantees that apply in the challenging finite-sample regime. We combine the design of targeted exploration with robust control by using tools from gain-scheduling. The designed controller guarantees robust closed-loop performance after an initial exploration phase. In the following sections, we discuss literature related to this thesis, followed by a sum- mary of contributions and the outline of the thesis. 3 Chapter 1. Introduction 1.2. Related work In this section, we provide an overview of research in the directions of system iden- tification analysis, learning for control, and dual control that are relevant to this thesis. 1.2.1. System identification analysis In what follows, we discuss literature on asymptotic and non-asymptotic analysis of system identification. These analyses form the theoretical foundation for under- standing how accurately system parameters can be estimated from data, and how the uncertainty in these estimates scales with factors such as data length, input de- sign, and noise characteristics. Classical system identification of linear systems with stochastic disturbances has a long history in the asymptotic regime [95]. The ellipsoidal uncertainty bound de- rived in [95] is an asymptotic confidence region around the estimated parameters. This confidence region is usually derived from the asymptotic covariance matrix of the estimator and is valid in the limit as the number of data samples tends to infin- ity. The shape and size of this ellipsoid are directly related to the Fisher information matrix [69]. In [142], a confidence region which is suitable for the design of robust control is provided directly in terms of the state-space parameters. Generally, the main focus of identification methods in the asymptotic regime has been the pro- vision of consistency guarantees, i.e., the convergence of the estimated parameters to the true parameters of the underlying system [14, 37, 88, 96]. These results are commonly achieved by ensuring that the inputs are persistently exciting [10]. Over the years, several works have focused on designing inputs that enhance and speed up parameter estimation. In this context, some early works focus on minimizing the estimation error and provide asymptotically exact expressions for the estima- tion error under various system and noise assumptions. These include cost-effective experiment design [21], structured system identification [75], and adaptive or op- timal experiment design for autoregressive models with exogenous inputs (ARX) [60, 135], autoregressivemoving-averagemodels with exogenous inputs (ARMAX) [77], and linear time-invariant (LTI) models under diverse noise assumptions [59, 134]. More recent results in the asymptotic regime that provide rigorous results on 4 1.2. Related work the estimation error can be found in [98, 153]. The earliest works on non-asymptotic analysis of system identification of linear systemswith stochastic disturbances provide lower bounds on the duration of iden- tification that guarantee desired worst-case error bounds [33, 112]. The first works in the setting of statistical learning provide high-probability finite-sample bounds on the predicted error in the parameters [28, 146, 154]. Spurred by advances in high-dimensional probability [144] and statistics [152], the results in [1] renewed interest in finite-sample identification for control by providing tighter uncertainty bounds and improved regret bounds. The results in [34] provide finite-sample un- certainty bounds suitable for robust control design, as well as the first end-to-end sample complexity guarantees for the infinite-horizon average-cost LQR problem. The results in [34], however, require independent data for identification. Following theworks in [1] and [34], a series of successiveworks provide improved uncertainty bounds for fully-observed stable and unstable systems [41, 42, 43, 82, 120, 130, 150]. In particular, the results in [130] are suitable when the data are not only dependent, but also affected by control inputs, unlike prior results in the setting of independent, or weakly dependent (mixing) data [28, 34, 154]. Beyond fully observable systems, some results for the more challenging problem of partially observable systems are provided by [92, 108, 121, 128, 139]. In the context of nonlinear settings, some for- mal system identification guarantees for different classes of nonlinear systems are provided in [56, 123]. Finite-sample error bounds for bilinear systems are provided in [31, 123]. Additionally, non-parametric frequency-domain finite-sample system identification results are provided in [67, 138], and overviews of finite-sample anal- ysis of system identification are presented in [141, 160]. In the context of linear systems with bounded disturbances, set-membership identification has been widely studied as a robust alternative to probabilistic meth- ods, offering guaranteed bounds onmodel uncertainty by characterizing all systems consistent with observed data and known disturbance bounds [19, 66, 76]. Data- dependent uncertainty bounds on the estimated parameters for linear systems with energy-bounded disturbances are provided in [53, 54]. These bounds naturally facilitate non-asymptotic analysis, as they rely on disturbances that are energy- bounded over finite time horizons. Various identification results that account for bounded noise can be found in [20, 106, 122, 127]. Such results have recently gained popularity for robust control design [18, 97, 148]. 5 Chapter 1. Introduction 1.2.2. Learning for control In model-based control, obtaining a suitable model of the underlying system from data is usually regarded as the most time-consuming and expensive task. The ac- curacy of the parameters significantly depends on the quality of the data used for system identification. Informative data can be strategically obtained from an experi- ment through the process of targeted exploration or optimal experiment design [62, 64, 113]. Specifically, targeted exploration inputs are tailored to reducemodel uncer- tainty, thereby ensuring the attainment of a desired accuracy in the identifiedmodel, or the feasibility of robust control design [63, 73, 74, 81, 151]. Preliminaryworks that account for model uncertainty are robust experiment design methods that propose iterative experiments [61, 74], or adaptive experiment design methods [13, 58, 60, 94]. Robust experiment design approaches that ensure desired quality constraints on the model while incurring minimal experiment cost were proposed in [21, 23, 24]. In particular, the earliest linear matrix inequality-based (LMI-based) frame- works for experiment design were proposed in [21, 81, 94]. Robust min-max exper- iment design approaches that account for prior uncertainty by utilizing an initial uncertainty set containing the true parameters are proposed in [68, 101, 114, 119]. However, these min-max experiment design approaches are intractable. Tractable robust experiment design approaches are proposed in [55, 86], albeit by utilizing approximations. In general, these works are heuristic and do not provide guaran- tees on excitation. Recent methods, such as [12, 22], outperform approaches that rely on conven- tional random exploration strategies and heuristics. The exploration methods pro- posed in [48, 78, 142] use inputs that consist of a linear state-feedback term and an additional Gaussian noise term for exploration. To tractably compute the pre- dicted uncertainty bound associatedwith the parameter estimates after exploration, the empirical covariance is approximated by the worst-case state covariance. This approximation fails to provide a priori guaranteed bounds on excitation of the ex- ploration inputs designed based on this approximation. The targeted exploration strategy designed in [48], which is inspired by results in [12] and based on the data-dependent uncertainty bound proposed in [34], is applicable in the finite-time regime. However, the uncertainty bounds on the model parameters in [34] require data to be independent, and hence not applicable to correlated time-series data. Re- 6 1.2. Related work cent works [122, 150] consider targeted exploration with periodic sinusoidal inputs. These works adopt a frequency-domain analysis for parameter estimation and en- sure non-asymptotic bounds on the estimated parameters. However, neither ap- proach explicitly accounts for transient effects during input design or analysis [95]. Hence, their guarantees are contingent on the dissipation of transient errors and the attainment of a steady-state response. This leads to prolonged experiment dura- tions, which may be impractical for many real-world applications. Typically, optimal experiment design decouples exploration from control. In con- trast, the field of adaptive control addresses simultaneous learning and control but has traditionally emphasized asymptotic stability [8, 51, 131, 157]. Recently, follow- ing works such as [2] and [35], there has been growing interest in adaptive LQR with non-asymptotic performance guarantees, such as regret bounds, to quantify learning efficiency over finite time horizons. In this setting, the objective is to mini- mize the regret, i.e., the difference between the average cost incurred by the learning controller and that of the optimal controller. Subsequent works provide computa- tionally more efficient algorithms towards achieving the optimal regret of O( √ T ) [2, 32, 36, 79, 100, 159], with [129] characterizing the min-max optimal regret with upper and lower bounds that are optimal in dimension and time horizon. A key principle underlyingmany of these results is optimism in the face of uncertainty, which guides exploration by favouring models that could lead to better performance [88]. Alternatively, works in [4, 107] and [3] utilize Thomson sampling for adaptive LQR, however, they propose computationally inefficient algorithms. 1.2.3. Dual Control The dual control paradigm, introduced in [44, 45, 46, 47], relies on stochastic DP which is computationally intractable. Either approximations of stochastic DP, or heuristic probing methods are typically adopted to solve the problem of tractability [50]. A detailed survey of dual control methods is provided in [50, 105]. Early works of implicit dual control methods, such as [133, 136, 137], involve ap- proximations of DP and are based on the wide-sense property, which requires lin- earization of the system dynamics and an approximation of the conditional proba- bility of the states by its mean and covariance [39]. These methods were extended to nonlinear systems with input constraints in [15], nonetheless based on some ap- 7 Chapter 1. Introduction proximations. Some other approaches approximate the dual control problem by utilizing a perturbation signal for exploration along with a robust controller [80], or modifying the loss function [51]. These works laid the foundation for balancing exploration with caution [11]. Explicit dual control methods use heuristic probing techniques for active learn- ing without the need to introduce approximations of DP [157]. These methods are closely related to optimal experiment design in closed loop [61, 74], as they utilize control inputs to regulate system dynamics and to probe the closed-loop system dy- namics by solving a combined problem. This led to application-oriented strategies for dual control which promote reducing uncertainty that would be beneficial for op- timizing cost [5]. Some recent application-oriented strategies are discussed in [9, 72, 89], however, they consider a special class of systems and their control strategies are not robust tomodel uncertainties. The family of coarse-IDmethods, based on the robust control framework System Level Synthesis (SLS), study robustness guarantees in system identification-based designmethods [34, 35, 36]. However, in these meth- ods, the control policies are not optimized to balance exploration and exploitation. A particularly appealing approach to the dual control problem is to sequentially apply some probing input for exploration, and then design a robustly stabilizing feedback based on the gathered data. Recent methods focus on targeted exploration and perform better thanmethods that use common greedy random exploration [12, 22, 24, 48, 78, 142]. In particular, these methods consider that exploration should be targeted in the sense that the resulting uncertainty reduction in themodel facilitates achieving a control goal and a performance objective. In [12], identification and ro- bust control are jointly designed. Iterations of identification and robust control de- sign are employed to ensure the uncertain parameters meet a specific accuracy level required for robust control. As a result, the desired properties for robust control can only be ensured after iterative experiments in [12]. Developing on [12], a dual control strategy is proposed in [48] that minimizes the worst-case cost achieved by a robust controller which is synthesized with reduced model uncertainty. How- ever, the uncertainty bounds utilized in [48] require data to be independent, and hence not applicable to time-series data available through exploration. In contrast, uncertainty bounds on the model parameters in [142] are directly applicable to cor- related time-series data. These bounds are used to design a targeted exploration strategy that excites the system to reduce uncertainty and specifically to improve a 8 1.3. Contributions and outline of the thesis robust LQR design in [142]. The approach in [78] extends the exploration strategy in [142] to a more realistic finite horizon problem setting that captures the trade- offs between exploration and exploitation better. While the resulting controllers in [12, 48, 78, 142] seem to perform well numerically, they lack the corresponding the- oretical performance guarantees. In particular, changes in the estimated mean of uncertain system parameters during the exploration phase are not accounted for in the methods in [48, 142]. In this thesis, we address the discussed limitations and simultaneously derive targeted exploration and robust control, yielding a tractable dual control approach with excitation and performance guarantees. 1.3. Contributions and outline of the thesis In this section, we provide an outline of the thesis and summarize its main contri- butions. Chapter 2: Preliminaries In Chapter 2, we provide a background on data-dependent uncertainty quantifi- cation, frequency-domain system identification, and performance metrics required for the main results in the thesis. In Section 2.1, we introduce asymptotic and non- asymptotic data-dependent uncertainty bounds on the parameter estimates, which form the basis for the design of targeted exploration and robust control. In Section 2.2, we provide preliminaries regarding the theory of spectral lines that are funda- mental for frequency-domain system identification. Finally, in Section 2.3, we sum- marize nominal and robust performance metrics that support the design of robust dual control. Chapter 3: Targeted exploration for linear systems In Chapter 3, we present a framework to design robust targeted exploration strate- gies for uncertain LTI systems subject to disturbances. In particular, we address chal- lenges associated with the type of disturbances affecting the system, i.e., stochastic or non-stochastic disturbances, and exploration time. We develop four different tar- geted exploration strategies tailored to specific disturbance models. 9 Chapter 1. Introduction Existing works that focus on targeted exploration use inputs in the form of a lin- ear state-feedback term and an additional Gaussian noise term for exploration [48, 78, 142]. These works utilize the data-dependent uncertainty bounds in [34, 142] and approximate the empirical covariance by the worst-case state covariance. This approximation fails to provide a priori guarantees of uncertainty bounds on the es- timated parameters. To address this limitation, we propose harmonic exploration inputs in the form of a linear combination of sinusoids of specific frequencies and optimized amplitudes, aiming to reduce uncertainty in a targeted fashion with the goal of guaranteed control performance. This choice is also supported in literature, where it was established that the robust optimal control input can be expressedwith appropriately chosen amplitudes and frequencies of the sinusoids [119]. In Section 3.1, we formulate the problem of targeted exploration. We specify the goal of targeted exploration as designing exploration inputs that yield informative data with minimal input energy, and, subsequently, parameter estimates with a de- sired accuracy. In Section 3.2, we address the setting of classical stochastic disturbances, i.e., inde- pendent, identically distributed (i.i.d.), and Gaussian with zero mean and known covariance. Building on the data-dependent uncertainty bound in Section 2.1.1, we derive sufficient conditions on the exploration data that guarantee the desired er- ror bound on the parameters estimated through exploration. We establish a lower bound on the finite excitation of the exploration inputs in terms of the spectral in- formation of the inputs [122]. The lower bound on excitation results in a bound on the uncertain system parameters, which can be leveraged in robust control design to provide performance guarantees. Furthermore, we robustly account for both ini- tial parametric uncertainty and the effect of the disturbances by deriving suitable bounds. Based on these results, we derive a semidefinite program-based (SDP- based) exploration strategy that ensures the desired error bound on the parameters. We demonstrate the effectiveness of our approach compared to random exploration strategies through a numerical example with a system that is ‘hard to learn’ [140]. While i.i.d. Gaussian disturbances are commonly assumed in classical and mod- ern control, reliable statistical information about the disturbances is often lacking in real-world systems. In such cases, energy-bounded disturbance models offer a practical alternative, with energy bounds derived directly from the physical prop- erties or operational constraints. Building on the data-dependent uncertainty bound 10 1.3. Contributions and outline of the thesis for energy-bounded disturbances [53], we design a targeted exploration strategy to estimate parameters with a desired error bound in Section 3.3. As a key contribu- tion, we derive sufficient conditions on the exploration data that ensure the desired closeness to the true parameters. Using common approximations [89], we derive an SDP-based design that allows us to compute an exploration strategy with min- imal energy. Through a numerical example, we demonstrate that the designed ex- ploration strategy for non-stochastic disturbances yields parameter estimates with a lower error in the presence of unmodeled system nonlinearities that are energy- bounded, in comparison to stochastic exploration strategies designed for classical stochastic noise. The results in Section 3.3 also serve as an intermediate step in de- riving a robust targeted exploration strategy for systems subject to energy-bounded disturbances. In Section 3.4, we leverage the sufficient conditions on the time-series exploration data to derive sufficient conditions directly on the inputs. In particular, we robustly account for parametric uncertainties and the effect of disturbances with suitable bounds. Leveraging these bounds along with the sufficient conditions, we derive exploration LMIs for an SDP-based targeted exploration design. Finally, based on a numerical example, we discuss the effect of the initial parametric uncertainty and the energy bound of the disturbances on the conservatism of the targeted explo- ration strategy. This example offers insights into the trade-offs between robustness and estimation accuracy. In Section 3.5, we present a non-asymptotic targeted exploration strategy for linear systems subject to stochastic disturbances based on the data-dependent uncertainty bound proposed in [1, 120]. Similar to Sections 3.2 and 3.4, we derive sufficient conditions directly in terms of the exploration inputs, and thereby ensure, for finite- time exploration, a desired parameter error bound with high probability. Unlike [122, 150], we explicitly account for the spectral transient error due to the input and the effect of the disturbances in the design of targeted exploration. Hence, the guar- antees on excitation hold in finite time and are not contingent on the attainment of a steady-state response. Through a numerical example, we demonstrate the practical impact of these transient effects on learning accuracy, underscoring the importance of careful input design to mitigate transient errors even when the exploration time is large. In summary, the main contributions of this chapter are the following: 11 Chapter 1. Introduction • Wepresent a targeted exploration strategy for systems subject to i.i.d. Gaussian disturbances with a priori asymptotic guarantees on the uncertainty bound of parameter estimates. • We present an approximate finite-time targeted exploration strategy for sys- tems subject to energy-bounded disturbances. • We present a targeted exploration strategy for systems subject to energy- bounded disturbances with a priori asymptotic guarantees on the uncertainty bound of parameter estimates. • We present a finite-time targeted exploration for systems subject to sub- Gaussian disturbances with a priori non-asymptotic high-probability guar- antees on the uncertainty bound of parameter estimates. The results of Chapter 3 have been previously presented in [JV1, JV3, JV4, JV5]. Chapter 4: Robust dual control In this chapter, we present a framework to design sequential robust dual controllers for uncertain LTI systems. In particular, we address the challenge of encapsulating the dual effect of performance improvement through the process of exploration and tailoring the exploration in amanner that is pertinent to performance improvement. In Section 4.1, we formulate the dual control problem of designing a robust con- troller that ensures closed-loop stability while satisfying performance specifications with high probability. We propose a sequential dual control approachwherein a tar- geted exploration phase is followed by the implementation of a parametrized state- feedback which achieves a desired performance. In order to tailor the exploration strategy to control objectives, we co-design the targeted exploration inputs and the robust controller based on the future model estimate and its associated uncertainty bound. Unlike prior dual control approaches such as [48, 78, 142], we specifically account for the changes in the model estimate through the process of exploration by utilizing tools from gain scheduling. In Section 4.2, we design the gain-scheduling controller. In order to account for the changes in the model estimate after exploration, we interpret the true system 12 1.3. Contributions and outline of the thesis as a linear parameter-varying (LPV) system, where the model estimate and associ- ated uncertainty bound after exploration are measured online. The new estimates are utilized as a scheduling variable, and tools from gain scheduling are utilized to design a robust controller that ensures stability and a desired performance. In Section 4.3, we first establish the relationship between the uncertainty bounds that influence the design of the gain-scheduling controller. We improve this re- lationship with less conservative joint-probabilistic bounds using parameter pro- jection. Finally, we present an SDP-based robust dual control problem that com- bines the design of the gain-scheduling controller and targeted exploration. Further- more, we provide a detailed theoretical analysis which confirms that the designed parametrized state-feedback controller ensures the desired performance after ex- ploration. In Section 4.4, we demonstrate the applicability of the proposed dual control ap- proach and discuss the trade-offs between exploration input energy and controller performance. In Section 4.5, we discuss themain features of the presented approach and its connections to other relevant methods. In summary, the main contribution of this chapter is a robust dual-control frame- work that integrates the design of targeted exploration inputs and a robust controller through gain scheduling. The results of Chapter 4 have been previously presented in [JV2, JV3]. Chapter 5: Conclusions In Chapter 5, we summarize and discuss the results of this thesis. Furthermore, we provide an outlook on interesting questions for future research. Appendix Appendix A contains the deferred technical proofs of some of the theoretical results in Chapters 2, 3, and 4, respectively. 13 Chapter 2. Preliminaries In this chapter, we present preliminaries for the thesis. We begin by introducing an uncertain discrete-time linear time-invariant (LTI) system subject to disturbances and outline data-driven uncertainty bounds on the parameter estimates based on the disturbances. Subsequently, we discuss preliminaries regarding the theory of spectral lines. Finally, we summarize results on nominal and robust performance analysis that form the basis for robust dual control design. 2.1. Data-dependent uncertainty quantification Data-dependent uncertainty quantification refers to the process of characterizing uncertainty in predicted model estimates based on the input and observed data, rather than solely relying on prior knowledge and assumptions. These quantifica- tion approaches leverage empirical covariance, spectral properties, or disturbance bounds to adapt and characterize uncertainty bounds on the estimates based on the specific dataset and context. This results in more accurate and less conservative uncertainty bounds that enable robust control synthesis and the design of targeted exploration strategies. In this section, we summarize prior results that provide data-dependent uncer- tainty bounds on the parameter estimates from [142], [53] and [1], respectively. We consider a discrete-time LTI system of the form xk+1 = Atrxk +Btruk + wk (2.1) where xk ∈ Rnx is the state, uk ∈ Rnu is the control input, and wk ∈ Rnx is the 15 Chapter 2. Preliminaries disturbance. It is assumed that the state can be measured and the initial state is at the origin, i.e., x0 = 0. The true values of the system parameters Atr, Btr, are initially uncertain. Henceforth, we denote ϕk = [ x⊤ k u⊤ k ]⊤ ∈ Rnϕ where nϕ = nx + nu. The system (2.1) can be re-written in terms of the uncertain parameter θtr = vec([Atr, Btr]) ∈ Rnθ as xk+1 = (ϕ⊤ k ⊗ Inx)θtr + wk (2.2) where nθ = nxnϕ. In what follows, we provide data-dependent uncertainty bounds on the parame- ter estimates based on the type of disturbances affecting the system. 2.1.1. Asymptotic uncertainty bound based on stochastic disturbances We present an asymptotic, data-dependent uncertainty bound on parameter esti- mates in the presence of stochastic disturbances [142]. Specifically, the disturbances affecting the system are assumed to be independent, identically distributed (i.i.d.), and Gaussian with zero mean and known covariance. This Gaussian disturbance assumption underpins much of classical and modern control theory due to mathe- matical tractability, optimality properties in estimation and control (e.g., in Kalman filtering [84], and linear-quadratic-Gaussian (LQG) control [103]), and its ability to approximate a wide range of real-world disturbance processes through the cen- tral limit theorem [52]. Leveraging this assumption, the empirical covariance of the observed data is used to characterize a confidence set of the estimated parameters. This confidence set enables the design of robust exploration and control strategies. We formalize the assumption on the disturbances as follows. Assumption 2.1. The disturbances wk are i.i.d. Gaussian distributed with zero mean and known covariance σ2 wInx , i.e., wk i.i.d.∼ N (0, σ2 wInx). In our setting, we also assume that some prior knowledge of the dynamics is avail- able. Assumption 2.2. The parameters θtr = vec([Atr, Btr]) have a Gaussian prior, i.e., θ ∼ N (θ̂0,Σθ,prior). Furthermore, there exists a matrixD0 ≻ 0 such that Σ−1 θ,prior = 1 σ2 w D0 ⊗ Inx . 16 2.1. Data-dependent uncertainty quantification The estimate θ̂T = vec([ÂT , B̂T ]) is the maximum a posteriori (MAP) estimate of the unknown parameters Atr and Btr, and is computed as: θ̂T = argmin θ T−1∑ k=0 1 σ2 w ∥∥(xk+1 − ([ x⊤ k u⊤ k ] ⊗ Inx ) θ) ∥∥2 + ∥∥∥θ − θ̂0 ∥∥∥ 2 Σ−1 θ,prior (2.3) with posterior covariance Σ−1 θ,post = Σ−1 θ,prior + 1 σ2 w DT ⊗ Inx (2.4) where DT = T−1∑ k=0 ϕkϕ ⊤ k . (2.5) Under Assumption 2.2, the posterior distribution p(θ|D) is given by N (θ̂T ,Σθ,post) [142, Prop. 2.1]. Remark 2.1. In case a prior as in Assumption 2.2 is not available, it can also be inferred from data. More precisely, given a data set D0 = {ϕk}−1 k=−T̄ obtained from a randomly exciting input, and a uniform prior over the parameters θ = vec([A,B]), i.e., p(θ) ∝ 1, the posterior distribution is given by N (θ̂,Σθ), where θ̂ = vec([Â, B̂]) is the ordinary least squares estimate, and Σ−1 θ = ( 1 σ2 w ∑−1 k=−T̄ ϕkϕ ⊤ k ) ⊗ Inx = 1 σ2 w D0 ⊗ Inx . This also justifies the structural assumption on Σ−1 θ,prior being of the form 1 σ2 w D0 ⊗ Inx for some D0 ≻ 0. Given observed dataDT+1 = {xk, uk}T−1 k=0 of length T+1, we are interested in quan- tifying the uncertainty associated with uncertain parameters θtr = vec([Atr, Btr]). The following lemma provides a high-probability credibility region for the uncer- tain parameters θtr under the assumption that T → ∞. Lemma 2.1. [142, Prop. 2.1, Lem. 3.1] Let Assumptions 2.1 and 2.2 hold. Given data set DT+1 of length T +1with estimate θ̂T = vec([ÂT , B̂T ]) (cf. (2.3)), and cδ = χ2 nxnϕ (δ)with 0 < δ < 1. Then, I. P(θtr = vec([Atr, Btr]) ∈ ΘT ) = 1− δ, where ΘT := { θ : (θ − θ̂T ) ⊤Σ−1 θ,post(θ − θ̂T ) ≤ cδ } (2.6) 17 Chapter 2. Preliminaries with Σ−1 θ,post = Σ−1 θ,prior + ( 1 σ2 w DT ) ⊗ Inx , and II. P([Atr, Btr] ∈ ∆T ) = 1− δ, where ∆T :=   A,B : [ (ÂT −A)⊤ (B̂T −B)⊤ ]⊤( 1 c̄ Dpost )[ (ÂT −A)⊤ (B̂T −B)⊤ ] ⪯ I    (2.7) with c̄ = cδσ 2 w and Dpost = D0 +DT . The result of Lemma 2.1 is a data-dependent uncertainty bound that can be uti- lized to synthesize robust controllers similar to approaches in [48, 78, 142]. Note that statement I implies statement II of Lemma 2.1 [142]. Initial estimates of the system parameters can be obtained from the mean of the prior distribution via vec([Â0, B̂0]) = θ̂0. The matrixD0 quantifies the robust bound associated with these initial estimates for a given probability 1− δ. More precisely, from Lemma 2.1, θtr ∈ Θ0 with probability 1− δ, where Θ0 := { θ : ( θ̂0 − θ )⊤(1 c̄ D0 ⊗ Inx )( θ̂0 − θ ) ≤ 1 } , (2.8) and [Atr, Btr] ∈ ∆0 with probability 1− δ where ∆0 :=   A,B : [ (Â0 − A)⊤ (B̂0 −B)⊤ ]⊤( 1 c̄ D0 )[ (Â0 − A)⊤ (B̂0 −B)⊤ ] ⪯ I    . (2.9) Denote ∆0 = [ Atr − Â0 Btr − B̂0 ] . (2.10) By applying the Schur complement twice on (2.9), Lemma 2.1 implies that P(∆⊤ 0 ∆0 ⪯ c̄D−1 0 ) = 1− δ. (2.11) Given observed data DT+1 = {ϕk}Tk=0 over T +1 time steps, the new estimates ÂT and B̂T can be computed from data DT+1 and the prior information as in (2.3). The matrixDpost := D0+DT quantifies the uncertainty associated with the estimates ÂT and B̂T . 18 2.1. Data-dependent uncertainty quantification 2.1.2. Non-asymptotic uncertainty bound based on energy-bounded disturbances We discuss a data-dependent uncertainty bound on the parameter estimates in the presence of energy-bounded disturbances [53]. In many practical scenarios, de- tailed statistical information about the disturbances affecting the system is rarely available or difficult to estimate reliably. Additionally, unmodeled dynamics or nonlinearities result in additional deterministic model mismatch that cannot be adequately captured by stochastic disturbance models [122]. In such cases, we can model disturbances as belonging to a known bounded set, such as point-wise bounds or energy bounds. Energy-bounded disturbance models thus offer a prac- tical alternative by avoiding reliance on statistical assumptions. We formalize the assumption on the disturbances as follows. Assumption 2.3. The disturbances wk are energy-bounded over a known time horizon T ∈ N>0, i.e., there exists a known constant γw > 0 such that T−1∑ k=0 ∥wk∥2 ≤ γw. (2.12) Given observed dataDT+1 = {xk, uk}Tk=0 of length T+1, the objective is to quantify the uncertainty associatedwith the uncertain parameters θtr. In order to simplify the exposition, we denote Φ = [ϕ0, . . . , ϕT−1] ∈ Rnϕ×T (2.13) and X⊤ = [x⊤ 1 , . . . , x ⊤ T ] ∈ R1×Tnx . (2.14) We obtain the following expressions for the estimate θ̂T = vec([ÂT , B̂T ]) and covari- ance P of the parameters from the standard least squares formulation [95, Section 1.3]: θ̂T = P T−1∑ k=0 (ϕ⊤ k ⊗ Inx) ⊤xk+1 = P (Φ⊗ Inx)X (2.15) 19 Chapter 2. Preliminaries and P = D−1 T ⊗ Inx (2.16) where DT = T−1∑ k=0 ϕkϕ ⊤ k = ΦΦ⊤. (2.17) The non-falsified region for the uncertain parameters θtr is provided in the follow- ing lemma. Lemma 2.2. [53] Let Assumption 2.3 hold. Given data set DT+1, the set of non-falsified parameters θ is given by ΘT := { θ : (θ − θ̂T ) ⊤P−1(θ − θ̂T ) ≤ G } (2.18) where G = γw + ∥θ̂T∥2P−1 −X⊤X. (2.19) Proof. The energy bound on the disturbances in (2.1) yields the following non- falsified set: ΘT = { θ : T−1∑ k=0 ∥xk+1 − (ϕ⊤ k ⊗ Inx)θ∥2 ≤ γw } (2.20) which can be equivalently written as θ⊤ ( (ΦΦ⊤)⊗ Inx ) θ − 2 ( X⊤(Φ⊤ ⊗ Inx) ) θ ≤ γw −X⊤X. (2.21) By adding ∥θ̂T∥2P−1 to both sides of (2.21), and using (2.15) and (2.16) to complete the squares, we get ∥θ − θ̂T∥2P−1 ≤ γw + ∥θ̂T∥2P−1 −X⊤X =: G, (2.22) which is equivalent to (2.18). 20 2.1. Data-dependent uncertainty quantification Given Assumption 2.3, the non-falsified setΘT provides an exact characterization of the set of parameters explaining the data. Similar non-falsified sets for matri- ces are considered in [18, 148]. The ellipsoid (2.18), derived from energy-bounded constraints, is characterized by a vector θ̂T and a matrix P , which correspond to the mean and covariance of the least squares estimator for linear systemswith Gaussian disturbances as in Lemma 2.1. However, unlike the case of zero mean i.i.d. Gaus- sian disturbances, in the case of energy-bounded disturbances the scaling G (2.19) of the bounding ellipsoid is also data-dependent. Furthermore, for γw ∝ T , we have G ≤ γw, and G scales at most linearly with T as T → ∞ [53, Lemma 4]. In contrast, the scaling of the confidence ellipsoid in the Gaussian case does not depend on T [95, 142]. Since P−1 increases linearly with T , the size of the confidence ellipsoid in the Gaussian case reduces with T [95, 142]. However, in the considered case of energy-bounded disturbances, the size of the non-falsified set ΘT , in general, does not decrease as T → ∞. 2.1.3. Non-asymptotic uncertainty bound based on stochastic disturbances We discuss a non-asymptotic data-dependent uncertainty bound on the parameter estimates under the assumption that the disturbances are sub-Gaussian [1]. The sub-Gaussian distributions encompass Gaussian, uniform, and other distributions with light tails. Specifically, sub-Gaussian randomvariables are characterized by ex- ponential tail decay. This enables the use of sharp concentration inequalities, such asHoeffding’s or Bernstein’s inequalities, to derive non-asymptotic high-probability bounds on the parameter estimates [102]. As a result, system identification analy- ses based on sub-Gaussian assumptions enable the design of finite-time targeted exploration strategies that guarantee learning while accommodating a wide range of disturbances that are beyond the strictly Gaussian case. We consider a filtration of σ-algebras {Fk}k≥0 such that ϕk isFk−1-measurable and wk is Fk-measurable. Definition 2.1. (Sub-Gaussian random vector [6, Definition 2]) A random vector wk ∈ Rnx with mean E[wk] = µ is called sub-Gaussian with variance proxy Σ ⪰ 0, i.e., wk ∼ 21 Chapter 2. Preliminaries subG(µ,Σ), if ∀λ ∈ Rnx , E [ exp(λ⊤(wk − µ))|Fk−1 ] ≤ exp (∥λ∥2Σ 2 ) . Assumption 2.4. The disturbance wk is conditionally sub-Gaussian with zero mean and known variance proxy σ2 wInx [1], i.e., wk ∼ subG(0, σ2 wInx). In our setting, we assume that some prior knowledge on the dynamics is available. Assumption 2.5. The parameters θtr = vec([Atr, Btr]) ∈ Rnθ lie in a known set Θ0, i.e., θtr ∈ Θ0, where Θ0 := { θ : (θ̂0 − θ)⊤(D0 ⊗ Inx)(θ̂0 − θ) ≤ 1 } , (2.23) with an estimate θ̂0 and for some D0 ≻ 0. Given observed data DT+1 = {xk, uk}Tk=0 of length T + 1, we focus on quantifying the uncertainty associated with the uncertain parameters θtr = vec([Atr, Btr]). For this purpose, we first outline regularized least squares estimation, and then summa- rize existing results by [1, 120] that provide a data-dependent uncertainty bound on the uncertain parameters. We denote Φ = [ϕ0, . . . , ϕT−1] ∈ Rnϕ×T , X = [x⊤ 1 , . . . , x ⊤ T ] ⊤ ∈ RTnx×1, W = [w⊤ 0 , . . . , w ⊤ T−1] ⊤ ∈ RTnx×1. (2.24) Given (2.24), the regularized least squares estimate is given by θ̂T = ((DT ⊗ Inx) + λInϕnx) −1 ︸ ︷︷ ︸ =:D̃−1 T (Φ⊗ Inx)X, (2.25) with the regularization constant λ > 0 and the excitation DT = ΦΦ⊤. (2.26) 22 2.1. Data-dependent uncertainty quantification Using (2.25), the estimate θ̂T can be expressed as θ̂T (2.2) = D̃−1 T (Φ⊗ Inx)((Φ ⊤ ⊗ Inx)θtr +W ) = D̃−1 T ((DT ⊗ Inx)θtr + (Φ⊗ Inx)W ). (2.27) Thus, the regularized least squares error satisfies θ̂T − θtr =− λD̃−1 T θtr + D̃−1 T (Φ⊗ Inx)W︸ ︷︷ ︸ =:ST . (2.28) Self-normalized Martingales: The sequence {ST}T≥0 with ST = (Φ ⊗ Inx)W is a martingale with respect to {FT}∞T=0 ((2.1), [156]). This sequence is crucial for the construction of confidence ellipsoids for θtr. The following theorem derives a ‘self- normalized bound’ for the martingale {ST}T≥0 by assuming scalar disturbances wk, i.e., nx = 1. Theorem 2.1. [1, Theorem 1](Self-normalized bound for vector-valued martingales) For any δ > 0, with probability at least 1− δ, for all T ∈ N>0, ∥ST∥2D̃−1 T ≤ 8σ2 w log ( 5nx det(D̃T ) 1 2 δ det(λInxnϕ ) 1 2 ) . (2.29) The following lemma generalizes the bound in Theorem 2.1 to vector-valued dis- turbances, i.e., nx > 1, by utilizing covering techniques as in [120, Proposition 8.2] or [141]. Lemma 2.3. (Self-normalized bound for vector processes) Let Assumption 2.4 hold. For any δ ∈ (0, 1), with probability at least 1− δ, for all T ∈ N>0, ∥ST∥2D̃−1 T ≤ 8σ2 w log ( 5nx det(D̃T ) 1 2 δ det(λInxnϕ ) 1 2 ) ︸ ︷︷ ︸ :=R(D̃T ) . (2.30) The proof of Lemma 2.3 is deferred to Appendix A.1.1. In order to derive the confidence ellipsoid, we first derive a bound on θtr of the form ∥θtr∥ ≤ θ̄. By applying the Schur complement twice to the condition in (2.23), we get (θ̂0 − θtr)(θ̂0 − θtr) ⊤ ⪯ 23 Chapter 2. Preliminaries (D0 ⊗ Inx) −1, from which we have ∥θ̂0 − θtr∥ ≤ ∥D− 1 2 0 ∥. (2.31) Using the triangle inequality, we have ∥θtr∥ ≤ ∥θ̂0∥+ ∥θ̂0 − θtr∥ (2.31) ≤ ∥θ̂0∥+ ∥D− 1 2 0 ∥ =: θ̄. (2.32) The following theorem utilizes the result in Lemma 2.3 to derive a data-dependent uncertainty bound in the form of a confidence ellipsoid centred at the estimate θ̂T for the uncertain parameters θtr. Theorem 2.2. [1, Theorem 2] Let Assumptions 2.4 and 2.5 hold. Then, for any δ > 0, P[θtr ∈ ΘT , ∀T ] ≥ 1− δ where ΘT = { θ : ∥θ̂T − θtr∥D̃T ≤ R(D̃T ) 1 2 + λ 1 2 θ̄ } . (2.33) The proof of Lemma 2.3 is provided inAppendixA.1.2. Similar to the utility of the data-dependent uncertainty bounds based on either energy-bounded disturbances or i.i.d. Gaussian disturbances, we can use this data-dependent uncertainty bound to design a finite-time targeted exploration strategy. 2.2. Frequency domain information using spectral lines In what follows, we discuss preliminaries regarding the theory of spectral lines, which deals with the analysis of frequency domain information that can be derived from time-series data [122]. The data-dependent uncertainty bounds in Lemmas 2.1 and 2.2, and Theorem 2.2 are based on the matrix DT , which is a quantitative measure of finite excitation defined as follows. Definition 2.2. (Finite Excitation [122]) A sequence {ϕk}k≥0,...T−1 is said to be finitely exciting from k = 0 to T − 1 if there exist constants 0 < ρ1 ≤ ρ2 such that ρ2I ⪰ T−1∑ k=0 ϕkϕ ⊤ k ︸ ︷︷ ︸ =DT ⪰ ρ1I. (2.34) 24 2.2. Frequency domain information using spectral lines In order to determine if the input signal is finitely exciting, we introduce the notion of a spectral line. Definition 2.3. (Sub-Gaussian Spectral Line [122]) A stochastic sequence {ϕk}T−1 k=0 is said to have a sub-Gaussian spectral line from k = 0 to T − 1 at a frequency ω ∈ ΩT := {0, 1/T, . . . , (T − 1)/T} with amplitude ϕ̄(ω) ∈ Cnϕ and variance proxy R2 T Inϕ ≻ 0 if 1 T T−1∑ k=0 ϕke −j2πωk − ϕ̄(ω) ∼ subG ( 0, R2 T Inϕ ) . (2.35) If noise is neglected, we recover a deterministic frequency component ϕ̄(ω) = 1 T T−1∑ k=0 ϕke −j2πωk. (2.36) The variance proxy R2 T Inϕ ≻ 0 in Definition 2.3 is a measure of the stochastic noise present in the sequence {ϕk}T−1 k=0 . The notion of a sub-Gaussian spectral line induces a requirement of appropriate behaviour over finite time. If the input has sufficiently many spectral lines, then the input signal is finitely exciting and can be used to pro- vide bounds for parameter estimation. In order to establish the relationship between the spectral content of the input signal and finite excitation, we utilize the expected information matrix which is defined as follows. Definition 2.4. (Expected Information Matrix [122]) Given a sequence {ϕk}T−1 k=0 with L sub-Gaussian spectral lines at frequencies ωi ∈ ΩT , i = 1, . . . , L with amplitudes {ϕ̄(ωi)}Li=1, the information matrix Φ̄ ∈ Cnϕ×L is defined as Φ̄ =   . . . ϕ̄(ω1) . . . ϕ̄(ωL) . . .   . (2.37) In deterministic system identification, estimation of unknownparameters ismade possible if Φ̄ has full rank and is numerically well conditioned. Since ωi ∈ ΩT , i = 1, . . . , L, we set T ≥ L and select L frequencies from ΩT . The spectral content in the expected information matrix can be used to determine whether a signal is finitely exciting or not [122]. In particular, a lower bound on finite excitation results in a 25 Chapter 2. Preliminaries bound on the uncertain parameters. This lower bound may be computed by us- ing the expected information matrix and can be leveraged in the design of targeted exploration and robust control. 2.3. Nominal and robust performance analysis We begin this section with a discussion on the criteria for nominal performance of linear dynamical systems. We consider the discrete-time linear time-invariant system xk+1 = Axk +Buk +Bwwk, zk = Cxk +Duk +Dwwk, (2.38) where xk ∈ Rnx is the state, uk ∈ Rnu is the input, wk ∈ Rnw is the disturbance, and zk ∈ Rnz is the output. We consider a stabilizing state-feedback controller uk = Kxk with K ∈ Rnu×nx . The closed-loop system can be written as xk+1 = (A+BK)xk +Bwwk, zk = (C +DK)xk +Dwwk. (2.39) We are interested in quantifying the effect of the disturbanceswk on the performance output zk, and thereby evaluating the performance of the closed-loop system. The performance specification is imposed on the channel w 7→ z. Let G(jω) = (C + DK)(ejωI − (A + BK))−1Bw + Dw denote the transfer function of the closed-loop system. There are several indicators for performance such as the worst-case gain, the energy in the impulse response of the system, and the asymptotic variance of the output. In what follows, we discuss criteria for quadratic, H∞ and H2 performance for the nominal model, assuming fixed, uncertainty-free parameters. 2.3.1. Nominal quadratic performance analysis In the following proposition, we characterize quadratic performance and stability of the closed-loop system in (2.39). 26 2.3. Nominal and robust performance analysis Proposition 2.1. (Quadratic performance, [125, Proposition 3.9]) The closed-loop system (2.39) with uk = Kxk achieves quadratic performance with index Pp = [ Qp Sp S⊤ p Rp ] with Rp ≻ 0, if (a) there exists an ϵ > 0 such that for initial state x0 = 0 and for all w ∈ ℓ2e, T̃∑ k=0 [ wk zk ]⊤ Pp [ wk zk ] ≤ −ϵ T̃∑ k=0 w⊤ k wk, ∀ T̃ ≥ 0 (2.40) holds, (b) for all ω ∈ R ∪ {±∞}, [ I T (jω) ]H Pp [ I T (jω) ] ≺ 0 (2.41) holds, (c) there exists X = X⊤ ≻ 0 such that   I 0 A+BK Bw 0 I C +DK Dw   ⊤   −X 0 0 0 0 X 0 0 0 0 Qp Sp 0 0 S⊤ p Rp     I 0 A+BK Bw 0 I C +DK Dw   ≺ 0, (2.42) where the statements (a), (b) and (c) are equivalent. The proposition characterizes quadratic performance with a frequency-domain inequality (2.41) and an LMI (2.42). 2.3.2. Nominal H∞ performance analysis TheH∞-performancemeasure provides a systematic framework to bound theworst- case amplification of disturbances in LTI systems. Rooted in frequency-domain anal- ysis, the H∞ norm of the transfer function of the closed-loop system quantifies the 27 Chapter 2. Preliminaries peak gain of the transfer function across all frequencies. The H∞ norm of G is de- fined as ∥G∥∞ := sup ω∈R σmax(G(jω)), (2.43) which is the supremum of the maximum singular value of the frequency response of the system. In the context of LTI systems as in (2.39), theH∞ norm ofG is equal to the ℓ2-gain of the closed-loop system, i.e., ∥G∥∞ = ∥G∥2,2. In the following proposi- tion, we characterizeH∞ performance of the closed-loop system in (2.39) as a special case of quadratic performance with Sp = 0, Rp = 1 γq I and Qp = −γqI . Proposition 2.2. (H∞ performance, [125, Proposition 3.12]) The closed-loop system (2.39) with uk = Kxk achieves the H∞-performance bound γp, if (a) ∥G∥∞ < γp, (b) for initial condition x0 = 0, the induced system norm ∥G∥2,2 := sup 0<∥w∥2<∞ ∥z∥2 ∥w∥2 < γp, (c) there exists an ϵ > 0 such that for initial state x0 = 0 and for all w ∈ ℓ2e, T̃∑ k=0 [ wk zk ]⊤ [−γpI 0 0 1 γp I ][ wk zk ] ≤ −ϵ T̃∑ k=0 w⊤ k wk, ∀ T̃ ≥ 0 (2.44) holds, (d) for all ω ∈ R ∪ {±∞}, G(jω)HG(jω) ≺ γ2 pI holds, (e) there exists X = X⊤ ≻ 0 such that   I 0 A+BK Bw 0 I C +DK Dw   ⊤   −X 0 0 0 0 X 0 0 0 0 −γpI 0 0 0 0 1 γp I     I 0 A+BK Bw 0 I C +DK Dw   ≺ 0, (2.45) where the statements (a), (b), (c), (d) and (e) are equivalent. 28 2.3. Nominal and robust performance analysis The proposition characterizesH∞ performancewith time-domain and frequency- domain inequalities, and with an LMI (2.45). 2.3.3. Nominal H2 performance analysis For the closed loop system (2.39)with transfer functionG(jω), theH2 norm satisfies ∥G∥22 = 1 2π ∫ π −π trace ( G(jω)G(jω)H ) dω. (2.46) Particularly in response to i.i.d. Gaussian noise inputs, i.e., wk i.i.d.∼ N (0, σ2 wI), the H2 norm provides a suitable stochastic interpretation in terms of the asymptotic out- put variance of the closed-loop system, i.e., limk→∞ E(∥zk∥2) [109]. The closed-loop system (2.39) with uk = Kxk and initial state x0 = 0 achieves the H2 performance bound γp if lim k→∞ E(∥zk∥2) < γ2 pσ 2 w. (2.47) In the following proposition, we characterize H2 performance of the closed-loop system in (2.39) in terms of LMIs. Proposition 2.3. (H2 performance, [125, Proposition 3.13]) The closed-loop system (2.39) with uk = Kxk and Dw = 0 achieves the H2 performance bound γp, if (a) the H2 norm satisfies ∥G∥2 < γp, (2.48) (b) there exist X = X⊤ ≻ 0 and Z = Z⊤ ≻ 0 such that [ −X + (A+BK)⊤X(A+BK) (A+BK)⊤XBw B⊤ wX(A+BK) B⊤ wXBw − γpI ] ≺ 0, [ X (C +DK)⊤ (C +DK) Z ] ≻ 0, trace(Z) < γp, where the statements (a) and (b) are equivalent. 29 Chapter 2. Preliminaries Note that the conditions in statement (b) in Proposition 2.3 can be written asX = X⊤ ≻ 0, Z = Z⊤ ≻ 0,   I 0 A+BK Bw 0 I   ⊤   −X 0 0 0 X 0 0 0 −γpI     I 0 A+BK Bw 0 I   ≺ 0, [ X (C +DK)⊤ (C +DK) Z ] ≻ 0, trace(Z) < γp. 2.3.4. Robust performance analysis Parametric uncertainties in the model can be either time-invariant or time-varying. Time-invariant uncertainties occur when the model is known up to some level of ac- curacy. Time-varying uncertainties occur when the uncertain model parameters are time-dependent, reflecting dynamic variations due to factors such as environmental changes or operational conditions. In this section, we consider a discrete-time linear parameter-varying (LPV) system xk+1 = A(δ)xk +B(δ)uk +Bw(δ)wk, zk = C(δ)xk +D(δ)uk +Dw(δ)wk, (2.49) where δ ∈ Rp represents the vector of uncertain parameters that affect the system matrices. We assume that δ belongs to an uncertainty set ∆ ⊆ Rp. We consider a stabilizing state-feedback controller uk = Kxk. The closed-loop system can be written as xk+1 = (A(δ) +B(δ)K)xk +Bw(δ)wk, zk = (C(δ) +D(δ)K)xk +Dw(δ)wk. (2.50) This representation allows for gain-scheduling robust control design procedures, where the controller parameters are adjusted online to account for the changing system dynamics [7, 93]. A detailed treatment of the robust controller design prob- lem for LPV systems with performance objectives can be found in [125, 126]. In the following proposition, we characterize robust quadratic performance and stability 30 2.3. Nominal and robust performance analysis of the closed-loop system in (2.50). Proposition 2.4. (Robust quadratic performance) The closed-loop system (2.50)with uk = Kxk achieves robust quadratic performance with index Pp = [ Qp Sp S⊤ p Rp ] and with Rp ≻ 0, if (a) there exists an ϵ > 0 such that for initial state x0 = 0, for all δ ∈ ∆, and for all w ∈ ℓ2e, T̃∑ k=0 [ wk zk ]⊤ Pp [ wk zk ] ≤ −ϵ T̃∑ k=0 w⊤ k wk, ∀ T̃ ≥ 0 (2.51) holds, (b) there exists X = X⊤ ≻ 0 such that ⋆⊤   −X 0 0 0 0 X 0 0 0 0 Qp Sp 0 0 S⊤ p Rp     I 0 A(δ) +B(δ)K Bw(δ) 0 I C(δ) +D(δ)K Dw(δ)   ︸ ︷︷ ︸ ⋆ ≺ 0, (2.52) for all δ ∈ ∆, where the statements (a) and (b) are equivalent. Similarly, criteria for other performance measures, such as H∞ and H2 perfor- mance (cf. Propositions 2.2 and 2.3), can be defined based on the closed-loop de- scription (2.50) for all δ ∈ ∆. The corresponding quadratic performance synthesis problem seeks to find con- troller parameters K and a symmetric common Lyapunov matrix X such that the family of LMIs (2.52) hold for all δ ∈ ∆. This approach can be conservative since it requires a single Lyapunovmatrix to certify performance for all uncertainties δ ∈ ∆. However, this approach is systematic and well-suited to convex optimization meth- ods like LMIs, making it a standard and practical tool in robust controller synthesis. 31 Chapter 3. Targeted exploration for linear systems In this chapter, we provide a framework to design targeted exploration strategies for uncertain discrete-time linear time-invariant (LTI) systems subject to disturbances. In particular, the proposed design methods address challenges associated with the type of disturbances affecting the system - stochastic or energy-bounded distur- bances, and exploration time. The goal of targeted exploration is to optimally ex- cite dynamical systems to reduce model uncertainty, thereby (i) attaining a model with desired accuracy, or (ii) ensuring the feasibility of a subsequent robust control design. In particular, in this chapter, we focus on designing targeted exploration strategies that ensure a desired accuracy on the estimated parameters after explo- ration. In what follows, we first introduce the problem of targeted exploration in Sec- tion 3.1 by presenting the setting, i.e., the model of the system considered. We then formulate the objective of targeted exploration and propose a class of input signals along with the overall exploration strategy. In Section 3.2, we consider clas- sical stochastic disturbances, i.e., Gaussian disturbances with zeromean and known covariance. We present a targeted exploration strategy with a priori asymptotic guarantees on parameter accuracy. In Section 3.3, we introduce energy-bounded disturbances and design a tractable finite-time exploration strategy, albeit without formal guarantees. We build on results provided in Section 3.3 by designing a ro- bust targeted exploration strategy for systems with energy-bounded disturbances with asymptotic guarantees on parameter accuracy in Section 3.4. Finally, in Section 33 Chapter 3. Targeted exploration for linear systems 3.5, we design a targeted exploration strategy for systems with sub-Gaussian distur- bances with high-probability non-asymptotic, finite-sample guarantees on parame- ter accuracy. This class of disturbances includes a wide range of disturbance mod- els, encompassing both bounded and non-Gaussian stochastic disturbances, thus broadening the applicability of the proposed approach. This chapter is based on and taken in parts literally from [JV1, JV3, JV4, JV5]. 3.1. Problem Statement We consider a discrete-time LTI dynamical system as in (2.1): xk+1 = Atrxk +Btruk + wk, where k ∈ N is the time, xk ∈ Rnx is the state, uk ∈ Rnu is the control input, and wk ∈ Rnx is the disturbance. The true system parametersAtr, Btr are initially uncertain. It is assumed the state xk is directly measurable. In order to simplify the exposition, we assume that the initial state x0 is zero. The disturbances are assumed to satisfy one of the following conditions: they are either stochastic, as in Assumptions 2.1 or 2.4, or energy-bounded, as in Assumption 2.3. Assumption 3.1. The system matrix Atr is Schur stable and the pair (Atr, Btr) is control- lable. 3.1.1. Exploration goal Since the true system parameters θtr = vec([Atr, Btr]) are not precisely known, there is a necessity to gather informative data through the process of exploration. Our primary goal is to design exploration inputs that excite the system, for a fixed user-chosen time T ∈ N>0, in a manner as to obtain an estimate θ̂T = vec([ÂT , B̂T ]) that satisfies (θtr − θ̂T ) ⊤ (Ddes ⊗ Inx) (θtr − θ̂T ) ≤ 1. (3.1) Here, Ddes ≻ 0 is a user-defined matrix characterizing how closely θ̂T approximates the true parameters θtr. The exploration inputs are computed such that it excites 34 3.1. Problem Statement the system sufficiently with minimal input energy, based on the initial parameter estimates. Denote U = [u⊤ 0 , · · · , u⊤ T−1] ⊤ ∈ RTnu . Bounding the input energy by a constant Tγ2 e ≥ 0 can be equivalently written as T−1∑ k=0 ∥uk∥2 = ∥U∥2 ≤ Tγ2 e . (3.2) This constraint prevents arbitrarily large exploration inputs, which, while poten- tially increasing the signal-to-noise ratio and yielding more informative data, could be expensive, unrealistic or unsafe in practical settings. 3.1.2. Exploration strategy We introduce a targeted exploration strategy in the form of multi-sine inputs with specified frequencies that explicitly shape themodel uncertainty, unlike greedy ran- dom exploration [48, 78, 142]. The exploration input sequence takes the form uk = L∑ i=1 ū(ωi) cos(2πωik), k = 0, . . . , T − 1, (3.3) where ū(ωi) ∈ Rnu are the amplitudes of the multi-sine inputs at L ∈ N>0 dis- tinct selected frequencies ωi ∈ ΩT = {0, 1/T, . . . , (T − 1)/T}. The amplitude of the spectral line of the sequence {uk}T−1 k=0 at frequency ωi is ū(ωi). Denote Ue = diag(ū(ω1), . . . , ū(ωL)) ∈ RLnu×L. By the Parseval-Plancheral identity, bounding the average input energy by a constant γ2 e (3.2) can be equivalently written as 1 T ∥U∥2 = L∑ i=1 ∥ū(ωi)∥2 = 1⊤ LU ⊤ e Ue1L ≤ γ2 e (3.4) where 1L ∈ RL×1 is a vector of ones, and the bound γe ≥ 0 is desired to be small. Using the Schur complement, this criterion is equivalent to the following LMI: Senergy-bound-1(γe, Ue) := [ γe 1⊤ LU ⊤ e Ue1L γeI ] ⪰ 0. (3.5) Remark 3.1. We require Atr to be Schur stable since we consider only open-loop inputs in 35 Chapter 3. Targeted exploration for linear systems our exploration strategy (3.3). Assumption 3.1 could be relaxed if an exploration input of the form in (3.3) with an additional linear feedback, i.e., vk = uk +Kxk, is utilized which ensures robust stability given an initial estimate and prior uncertainty (cf. Assumptions 2.2, 3.3 or 2.5). In order to achieve the exploration goal, the amplitudes of the sinusoidal explo- ration inputs need to be optimized such that by applying the exploration inputs, the obtained estimate satisfies the desired uncertainty bound (3.1). To this end, in the following sections, we develop targeted exploration methods tailored to the nature of the disturbances affecting the system. 3.2. Asymptotic targeted exploration for systems with stochastic disturbances In this section, we propose a targeted exploration strategy for systems subject to classical stochastic disturbances, specifically, i.i.d. Gaussian disturbances with zero mean and known covariance as in Assumption 2.1. The strategy is based on the data-dependent uncertainty bound provided in Lemma 2.1 and provides asymp- totic guarantees. First, we provide sufficient conditions on the exploration data that guarantee a desired error bound on the estimated parameters through targeted ex- ploration in Section 3.2.1. We then derive a lower bound on finite excitation of the exploration data using the spectral information of the exploration inputs in Section 3.2.2. Since this bound is non-convex in the decision variables and depends on un- certain model parameters, a convex relaxation procedure is carried out in Section 3.2.3, and required bounds on the effect of the model uncertainty are derived in Sec- tion 3.2.4. Finally, in Section 3.2.5, we obtain an SDP for exploration which provides us with exploration inputs that guarantee the desired error bound on the estimates. In Section 3.2.6, we demonstrate the applicability of the targeted exploration strat- egy through a numerical example. In Section 3.2.7, we discuss the main features of the presented approach and its connections to existing works. 36 3.2. Asymptotic targeted exploration for systems with stochastic disturbances 3.2.1. Sufficient conditions for targeted exploration Given the form of the exploration inputs in (3.3), the exploration goal (3.1), and the data-dependent uncertainty bound in Lemma 2.1, we derive conditions that the ex- ploration data have to satisfy to achieve the exploration goal in the following propo- sition. Proposition 3.1. Suppose ϕk satisfies T−1∑ k=0 ϕkϕ ⊤ k +D0 − c̄Ddes ⪰ 0. (3.6) Then, the estimate θ̂T computed as in (2.3) satisfies the exploration goal (3.1) with proba- bility at least 1− δ. Proof. The bound in (2.6) of Lemma 2.1 can be written as (θ − θ̂T ) ⊤ (1 c̄ (D0 +DT )⊗ Inx ) (θ − θ̂T ) ≤ 1. (3.7) By applying the Schur complement twice to (3.7), we get (θ − θ̂T )(θ − θ̂T ) ⊤ ⪯ c̄(D0 +DT ) −1 ⊗ Inx (2.5) = c̄ ( D0 + T−1∑ k=0 ϕkϕ ⊤ k )−1 ⊗ Inx (3.8) with probability 1− δ. From (3.6), we get ( T−1∑ k=0 ϕkϕ ⊤ k +D0 )−1 ⊗ Inx ⪯ 1 c̄ D−1 des ⊗ Inx . (3.9) Furthermore, by inserting (3.9) in (3.8), we get (θ − θ̂T )(θ − θ̂T ) ⊤ ⪯ (Ddes ⊗ Inx) −1. (3.10) Finally, applying the Schur complement twice to (3.10) yields the exploration goal (3.1). Note that Inequality (3.6) depends on ϕk quadratically, which further depends 37 Chapter 3. Targeted exploration for linear systems on the exploration inputs Ue and disturbances wk. Furthermore, the linear mapping from the input sequence to the state sequence is unknown since Atr and Btr are unknown. In the following sections, we address these challenges. 3.2.2. Bound on finite excitation based on spectral lines For the system evolving under the exploration input as given in (3.3), the uncer- tainty bound on the parameters can be computed from the expected information matrix Φ̄ of the input (cf. (2.37)). As a prerequisite to computing the uncertainty bound, it is necessary to establish the relationship between the spectral content of the observed state xk and the applied input uk. Given uk as in (3.3), xk hasL spectral lines from 0 to T − 1 at distinct frequencies ωi ∈ ΩT , i = 1, . . . , L with amplitudes: x̄(ωi) = (ej2πωiI − Atr) −1Btr︸ ︷︷ ︸ =:Vx,i ū(ωi) + (ej2πωiI − Atr) −1 ︸ ︷︷ ︸ =:Yx,i w̄(ωi) + x̄err(ωi) (3.11) where ū(ωi) and w̄(ωi) are the amplitudes of the spectral lines of the input uk and disturbances wk at a frequency ωi, respectively. The transient error in the amplitude of a spectral line x̄err(ωi) decays uniformly (with rate 1√ T ) to 0 as T → ∞ (cf. As- sumption 3.1, [95, Theorem 2.1]). To simplify the exposition, first, we will assume that the transient error can be neglected. Assumption 3.2. The transient error satisfies x̄err(ωi) = 0 for all ωi ∈ ΩT . Remark 3.2. Assumption 3.2 holds naturally if we let T → ∞. A detailed treatment of the transient error term is provided later in Section 3.5 where Assumption 3.2 can be relaxed. Furthermore, ϕk has L spectral lines from 0 to T − 1 at distinct frequencies ωi ∈ ΩT , i = 1, . . . , L with amplitudes ϕ̄(ωi) := [ Vx,i Inu ] ︸ ︷︷ ︸ =:Vϕ,i ū(ωi) + [ Yx,i 0 ] ︸ ︷︷ ︸ =:Yϕ,i w̄(ωi). (3.12) We compactly define Φ̄ = [ϕ̄(ω1), . . . , ϕ̄(ωL)] ∈ Cnϕ×L, (3.13) 38 3.2. Asymptotic targeted exploration for systems with stochastic disturbances which satisfies Φ̄ = Vϕ,trUe︸ ︷︷ ︸ =:Φ̄u +Yϕ,trW︸ ︷︷ ︸ =:Φ̄w , (3.14) with Vϕ,tr := [Vϕ,1, · · · , Vϕ,L] ∈ Cnϕ×nuL, Yϕ,tr := [Yϕ,i, · · · , Yϕ,L] ∈ Cnϕ×nxL W := diag(w̄(ω1), . . . , w̄(ωL)) ∈ CnxL×L. (3.15) The effect of the disturbances wk is captured by W . Each block w̄(ωi), i = 1, ..., L, on the diagonal ofW is Gaussian (cf. Assumption 2.1), i.e., w̄(ωi) ∼ N (0, σ2 w̄I)with σ2 w̄ = σ2 w T , since the DFT of a Gaussian signal is also Gaussian, but with a distinct variance as derived in Appendix A.2.1. The following lemma presents a clear rela- tionship between the spectral content of the signal and finite excitation. Lemma 3.1. For any ϵ ∈ (0, 1), ϕk satisfies T−1∑ k=0 ϕkϕ ⊤ k ⪰ T L ( (1− ϵ)Φ̄uΦ̄ H u − ( 1−ϵ ϵ ) Φ̄wΦ̄ H w ) . (3.16) Proof. Note that for any unit vector z ∈ Cnϕ , and any realization {ϕk}T−1 k=0 , zH ( 1 T T−1∑ k=0 ϕkϕ ⊤ k ) z = 1 T T−1∑ k=0 ∣∣ϕ⊤ k z ∣∣2 = 1 T T−1∑ k=0 ∣∣ϕ⊤ k z ∣∣2 · 1 L L∑ l=1 ∣∣e−j2πωlk ∣∣2 ︸ ︷︷ ︸ =1 ≥ 1 L L∑ l=1 ∣∣∣∣∣ 1 T T−1∑ k=0 ϕ⊤ k ze −j2πωlk ∣∣∣∣∣ 2 , (3.17) by Jensen’s inequality. This leads to zH ( 1 T T−1∑ k=0 ϕkϕ ⊤ k ) z ≥ 1 L L∑ l=1 ∣∣∣∣∣ 1 T T−1∑ k=0 ϕ⊤ k ze −j2πωlk ∣∣∣∣∣ 2 (3.14) = 1 L ( zH(Φ̄u + Φ̄w)(Φ̄u + Φ̄w) Hz ) . (3.18) 39 Chapter 3. Targeted exploration for linear systems By Young’s inequality [29], for any ϵ > 0, we have Φ̄uΦ̄ H w + Φ̄wΦ̄ H u ⪰ −ϵΦ̄uΦ̄ H u − 1 ϵ Φ̄wΦ̄ H w (3.19) and hence (Φ̄u + Φ̄w)(Φ̄u + Φ̄w) H ⪰ (1− ϵ)Φ̄uΦ̄ H u − ( 1− ϵ ϵ ) Φ̄wΦ̄ H w. (3.20) By inserting Inequality (3.20) in Inequality (3.18), we get (3.16). From Lemma 3.1, by using (3.14) in (3.16), for any ϵ ∈ (0, 1), ϕk satisfies DT = T−1∑ k=0 ϕkϕ ⊤ k ⪰ T L ( (1− ϵ)Vϕ,trUeU ⊤ e Vϕ,tr H − ( 1−ϵ ϵ ) Φ̄wΦ̄ H w ) . (3.21) Inequality (3.21) allows us to determine a lower bound on the finite excitation Dpost = D0 + DT , and thereby, an upper bound on the uncertainty of the MAP estimate using Lemma 3.1. Note that this lower bound depends on the amplitudes of the exploration inputs Ue (cf. (2.37), (3.14)), as well as on the effect of the dis- turbances. However, determining a lower bound on DT based on Inequality (3.21) results in non-convex constraints in the decision variable Ue. We circumvent this issue by using a convex relaxation procedure. 3.2.3. Convex relaxation The following lemma provides a lower bound on DT which is linear in Ue. Lemma 3.2. For any matrices Ũ ∈ RLnu×L and Ue ∈ RLnu×L, and any ϵ ∈ (0, 1), we have: DT ⪰T L ( (1− ϵ) ( Vϕ,tr ( UeŨ ⊤ + ŨU⊤ e − Ũ Ũ⊤ ) V H ϕ,tr ) − ( 1−ϵ ϵ ) Φ̄wΦ̄ H w ) . Proof. We have Vϕ,trUeU ⊤ e V H ϕ,tr − Vϕ,trUeŨ ⊤V H ϕ,tr − Vϕ,trŨU⊤ e V H ϕ,tr + Vϕ,trŨ Ũ⊤V H ϕ,tr =Vϕ,tr(Ue − Ũ)(Ue − Ũ)⊤V H tr ⪰ 0 (3.22) 40 3.2. Asymptotic targeted exploration for systems with stochastic disturbances and hence Vϕ,trUeU ⊤ e V H ϕ,tr ⪰ Vϕ,tr ( UeŨ ⊤ + ŨU⊤ e − Ũ Ũ⊤ ) V H ϕ,tr. (3.23) Inserting Inequality (3.23) in Inequality (3.21) leads to (3.22). The bound derived in Lemma 3.2 is tight in case Ũ = Ue. However, since Ue is unknown, we consider a candidate Ũ of linearly independent amplitudes corre- sponding to their respective spectral lines. We later embed this relaxation in an iterative process to reduce conservatism. Furthermore, in (3.22), Vϕ,tr and Yϕ,tr (in Φ̄w = Yϕ,trW) are unknown. Hence, in what follows, suitable bounds are derived. 3.2.4. Bounds on transfer matrices Denote Ṽϕ = Vϕ,tr − V̂ϕ (3.24) where the estimate V̂ϕ = [V̂ϕ,1, · · · , V̂ϕ,L] ∈ Cnϕ×Lnu (3.25) is computed using the prior Â0 and B̂0 as in Assumption 2.2. We show how to com- pute matrices Γ̃ϕ ≻ 0 and Γϕ ≻ 0 such that ṼϕṼ H ϕ ⪯ Γ̃ϕ, and Yϕ,trY H ϕ,tr ⪯ Γϕ, with ∥Yϕ,tr∥ ≤ √ ∥Γϕ∥ =: γϕ,y, (3.26) assuming θtr ∈ Θ0 (Assumption 2.2) using robust control tools in Appendices A.2.2 and A.2.3, and using scenario optimization in Appendix A.2.5. Utilizing (3.26), we can derive a bound on Φ̄w of the form ∥Φ̄w∥ ≤ ∥Yϕ,tr∥∥W∥. A bound on W can be determined since each block on the diagonal of W is Gaussian if Assumption 2.1 holds. In particular, we have ∥W∥ = max i=1,...,L ∥w̄(ωi)∥. (3.27) 41 Chapter 3. Targeted exploration for linear systems Since w̄(ωi) ∼ N (0, σ2 w̄I), we have that ∥w̄(ωi)∥2 ∼ σ2 w̄χ 2 nx , and hence P ( ∥w̄(ωi)∥2 ≤ l21 ) = 1− δ, ∀i = 1, ..., L (3.28) with l21 = σ2 w̄χ 2 nx (1− δ). Therefore, we get P(∥W∥ ≤ l1) = 1− δ. (3.29) Using (3.26) and (3.29), we derive Φ̄wΦ̄ H w ⪯ l2I := (γϕ,yl1) 2I. (3.30) The following lemma provides joint probabilistic bounds on Ṽϕ, Yϕ,tr and W . Lemma 3.3. Let Assumption 2.2 hold. Then P((ṼϕṼ H ϕ ⪯ Γ̃ϕ) ∩ (∥Yϕ,tr∥ ≤ γϕ,y) ∩ (∥W∥ ≤ l1)) ≥ 1− 2δ. (3.31) Proof. From Lemma 2.1, since P(θtr ∈ Θ0) = 1 − δ, the bounds in (3.26) hold with probability 1− δ: P((ṼϕṼ H ϕ ⪯ Γ̃ϕ) ∩ (∥Yϕ,tr∥ ≤ γϕ,y)) ≥ P(θtr ∈ Θ0) = 1− δ. Hence, P((ṼϕṼ H ϕ ⪯ Γ̃ϕ) ∩ (∥Yϕ,tr∥ ≤ γϕ,y) ∩ (∥W∥ ≤ l1)) ≥ P((θtr ∈ Θ0) ∩ (∥W∥ ≤ l1)) ≥ 1− P(θtr /∈ Θ0)− P(∥W∥ ≰ l1) (3.29) ≥ 1− 2δ, (3.32) wherein the penultimate inequality follows from De Morgan’s law. 3.2.5. Final bound on the informativity of exploration In the following theorem, we compute a lower boundDT on the excitationDT which yields an upper bound on the empirical covariance of the exploration data, before the process of exploration. 42 3.2. Asymptotic targeted exploration for systems with stochastic disturbances Theorem 3.1. Let Assumption 2.2 hold. Suppose there exists matrix Ue and τ ≥ 0 such that Sexploration(ϵ, τ, Ũ , Ue, V̂ϕ, l, DT , Γ̃ϕ) :=[ (1− ϵ) (U⊤ e Ũ + Ũ⊤Ue − Ũ⊤Ũ) 0 0 − ( 1−ϵ ϵ ) l2I − L T DT ] − τ [ −I V̂ H ϕ V̂ϕ Γ̃ϕ − V̂ϕV̂ H ϕ ] ⪰ 0 (3.33) withDT = c̄Ddes−D0. Then, the application of the input (3.3) implies that with probability at least 1− 2δ: DT ⪰ DT , (3.34) ensuring the exploration goal (3.1). Proof. By using (3.24) and (3.26), we get Vϕ,trV H ϕ,tr − Vϕ,trV̂ H ϕ − V̂ϕV H ϕ,tr + V̂ϕV̂ H ϕ ⪯ Γ̃ϕ, which can be written as [ V H ϕ,tr I ]H [ −I V̂ H ϕ V̂ϕ Γ̃ϕ − V̂ϕV̂ H ϕ ][ V H ϕ,tr I ] ⪰ 0. (3.35) Additionally, we have Φ̄wΦ̄ H w ⪯ l2I as in (3.30). From Lemma 3.2, and by using (3.30), the following inequality implies the condition (3.6) in Proposition 3.1, i.e., DT ⪰ DT = c̄Ddes +D0 : (1− ϵ) ( Vϕ,tr ( UeŨ ⊤ + ŨU⊤ e − Ũ Ũ⊤ ) V H ϕ,tr ) − ( 1− ϵ ϵ ) l2I − L T DT ⪰ 0. (3.36) Furthermore, (3.36) can be written as [ ∗ ∗ ]H [ (1− ϵ)(UeŨ ⊤ + ŨU⊤ e − Ũ Ũ⊤) 0 0 − ( 1−ϵ ϵ ) l2I − L T DT ][ V H ϕ,tr I ] ⪰ 0. (3.37) From Lemma 3.3, Inequalities (3.35) and (3.30) hold jointly with probability 1− 2δ. 43 Chapter 3. Targeted exploration for linear systems By using the matrix S-lemma [26, 149], (3.36) holds for all Vϕ,tr satisfying (3.35) if (3.33) holds with τ ≥ 0. Hence, if there exists Ue sa