This course provides a broad overview of ofstate-of-the-art parallel and distributed machine learning (ML) and deep learning (DL) algorithms and systems, with a strong focus on the scalability, resource efficiency, data requirements, and robustness of the solutions. This course covers effective ways to map state-of-the-art ML and DL solutions to parallel AI accelerators such as GPUs and TPUs. A set of techniques are presented to efficiently scale ML and DL workloads to a large number of distributed machines in the presence of system failures and malicious attacks. Finally, methods for improving the scalability and efficiency of generative learning and graph learning approaches are covered.