دانلود کتاب روش جهت متناوب ضرایب برای یادگیری ماشین

Alternating direction method of multipliers (ADMM) is a magic algorithm to me. In my biased opinion, it is more or less a universal method for solving a wide range of constrained problems that ordinary practitioners in machine learning may encounter. Unlike gradient descent, which is roughly a universal method for unconstrained problems, ADMM appears to be more elegant yet less transparent. The secret lies in the Lagrange multiplier, which temporarily makes the constrained problem unconstrained, not only removing the difficulty in handling the constraints but also overcoming some inherent defects of the penalty method and the projected gradient descent, while non-experts are much easier to think of the latter two methods. The Lagrange multiplier also plays a central role in the proofs of convergence and convergence rate of ADMM. With possible exaggeration, I would say that one who cannot appreciate the beauty of ADMMcannot be a good researcher in optimization. Since my first encounter with ADMM around 2009, I have seen that more and more machine learning people are using ADMM and extending its scope of applications. I also benefited a lot from and contributed a bit to the new developments. Yet, I also found that many engineers are not using ADMMcorrectly (the most notable example is to naively apply the ADMMfor two blocks of variables to problems with more than two blocks). There has been an excellent tutorial on ADMM, Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers, written by Boyd et al. in 2011. Nonetheless, it is now 10 years old and does not cover the new developments, which I actually think are more important than the traditional ones for the wider applications of ADMM, because the new developments were done out of demands from real applications in signal processing and machine learning. So, I think that writing a new book on ADMM will be very useful for many people, including myself when teaching and advising students. My goal is to incorporate the most important aspects of the new developments in ADMM, rather than being confined to the traditional materials, which are typically for convex and two-block cases. Clearly, I am unable to review all papers on ADMM. So, the strategy is to choose representative algorithms by their types (e.g., convex, nonconvex, stochastic, and distributed) instead. As a result, one should not be surprised that some variants of ADMM are not included (because they are not the most representative ones of their types but just discuss in more depth, or are too complex to use or analyze) while some variants of ADMM appear to be rather crude but they are still included (because that type of ADMM is less explored). Of course, personal flavor and limited knowledge also matter greatly. Finally, being self-contained is also very important. So, I also want to present proofs in detail.