دانلود کتاب مهندسی داده با GCP

With enterprises increasingly adopting a hybrid and multi-cloud strategy, Internet of Things (IoT) and artificial intelligence to innovate and stay ahead of its competitors, the variety, veracity and volume of data has exponentially increased. Building a scalable enterprise data platform has increasingly become complex and a challenge. This book is designed to serve as a comprehensive and practical guide to building enterprise scale data platform in Google Cloud Platform (GCP) leveraging its innovative, serverless managed data services. The book is structured to start with the key foundation concepts of data engineering. It covers a wide range of GCP data engineering services for data ingestion, data storage, data warehouse and data transformation. The book also explains how the curated data can be used for analytics and machine learning. Finally, the book also covers advanced topics such as data migration, data sharing and emerging trends in the discipline of data engineering in GCP. Throughout the book, you will learn the key capabilities and configuration of the various data services. You will also learn how to design these services providing enterprise grade reliability, scalability and security. The various chapters in this book also provide hands-on practical use cases and examples to provide a practical perspective apart from the theoretical concepts. This book is intended for beginners both students and professionals who have an ambition to make data engineering as their career in roles. The book is also designed for experienced data engineer professionals who want to expand their knowledge and improve their skills in data engineering in GCP. With this book, you will gain the knowledge and skills to become a proficient and skilled data engineer and data architect. Machine learning engineers and data scientists will also find this book very informative to understand data engineering in GCP to be efficient in their roles. Chapter 1: Foundations of Data Engineering – Introduces the various important concepts of data engineering. This chapter explains the end to end process from data ingestion, data storage to transformation and processing including the conecpts of ETL and ELT. Furthermore, this chapter also gives the reader an overview of designing scalable, reliable and secure data engineering solutions. Chapter 2: Data Engineering Services in GCP – This chapter introduces the various components and services available in GCP for data engineering and explains why it is the leading cloud platform for data engineering by comparing its capabilities with the other leading public clouds. This chapter also walks through the GCP environment setup which will be needed in further chapters for hand-on exercises. This environment setup will include details on how Google Cloud services are logically structured in projects, the user access and management as well as core networking concepts which the reader should be aware of to make effective use of the concepts covered in the rest of the chapters. Chapter 3: BigQuery Data Warehousing Service – In this chapter, the reader is introduced to the concept of data warehousing and how it is different from normal transactional databases. The chapter then introduces the BigQuery service and explains the BigQuery console in the Cloud console. Further, it explains all the features available in the console to import, store and query data and its integration capabilities. Also, it covers architecture, key best practices and standards for high performance. Finally, it provides hands-on step by step exercise creating and using the BigQuery service. Chapter 4: Data Ingestion Using Pub/Sub and Dataflow – This chapter explains the various concepts involved in data ingestion while building a data processing pipeline. It introduces Pub/Sub service and Dataflow service and explains the details on how to configure these services for various scenarios/use case. It also explains in detail the concept of change data capture which is an important concept for real time stream processing. It explains the data stream service and how it can be configured for CDC and used in a Dataflow. Finally, the chapter provides hands-on exercise on creating an end to end data pipeline in Dataflow using Apache Beam. Chapter 5: ETL and Orchestration Using Cloud Composer – In this chapter, the reader is introduced on data pipeline orchestration and why it is an important consideration in setting up complex data pipelines. It then introduces Apache Airflow for orchestration and different ways to run Apache Airflow in GCP. Furthermore, it explains cloud composer service in GCP for running and scheduling Apache Airflow. Finally, it provides step by step guidance on creating a multi-step pipeline in GCP to achieve the desired outcomes. Chapter 6: Data Lakes Using Cloud Storage and Dataproc – Introduces the concept of data lake. It also compares data lakes and data warehouses and why they complement each other in an enterprise. It explains the various GCP services involved in data transmission, processing and storage. For data storage, it explains the details of the Cloud Storage and its configuration for storing the transferred data.It then introduces the Dataproc GCP service for the data processing. It explains how Apache Hadoop and Spark Clusters can be configured and spun for large scale processing of raw data. Finally, it goes through the best practices and how the costs can be managed and optimized in real-world implementations for data processing solutions. Chapter 7: Data Visualization Using BigQuery and Looker – Provides an overview of data visualization concepts and why it is important. It also explains the various options available in GCP for data visualization. The chapter then provides an overview of Looker studio and Looker as a tool for data visualization. It explains the difference between Looker studio and Looker and how the visualization reports and dashboards can be configured in these services. Further, it explains the best practices and strategies involved in exposing the data in BigQuery for visualization in Looker. Finally, it walks through the step by step process in creating a report with sample data in BigQuery. Chapter 8: Data Migration Using Database Migration Service – This chapter introduces the concept of data migration in the cloud and why having a robust database migration service in any cloud is very important. It then introduces the GCP database migration service which is used for migrating data from either on-premise or other cloud databases. It also provides detail of configuring database migration for common database services such as PostgreSQL, MySQL, Oracle, etc., to GCP cloud database services along with key best practices. The chapter also provides an overview of pricing for these services and various considerations when migrating data to the database in GCP. Chapter 9: Data Integration and Machine Learning Pipelines in GCP – This chapter introduces the various options available in GCP to connect to external systems and transfer data into GCP for further processing in GCP. It provides details on the BigQuery Data Transfer Service and the various configuration and capabilities available in this service to transfer data from on-premise, private cloud and other public Cloud Storage services. This chapter provides an overview of the Vertex AI platform in GCP. It also provides details on integrating data with this AI platform. Further, it provides how ML models can be prepared and used for training them. Finally, it provides a hands-on exercise on creating a ML pipeline for data preparation and training machine learning models. Chapter 10: Cloud Monitoring, DevOps Automation and Best Practices – Introduces the concepts of cloud logging and monitoring and why this is important for complex data pipelines. It provides information to the readers on the various GCP services involved in logging and monitoring such as cloud monitoring and how it can be configured for monitoring data pipelines in GCP. The chapter also introduces CI/CD automation and the GCP services available for configuring CI/CD pipelines for deployment automation. This chapter also talks about various cost optimization strategies and best practices to follow in general in data engineering in GCP. Finally it provides the data governance strategies and policies which are typically followed to ensure data security and data integrity. Chapter 11: Data Exchange and Sharing Using BigQuery Sharing – This chapter covers the approach and capabilities in GCP for sharing/exchanging data between organizations efficiently and securely. In this context, it provides an overview of the GCP BigQuery Sharing (earlier Analytics Hub) service and introduces the key features available in this service. It explains why this is critical in today’s connected world with multiple enterprises working closely together as partners and explains the configuration of this service. Finally, the chapter also covers the pricing and key best practices to be followed in configuring this service for the customers. Chapter 12: Emerging Trends and Real-world Use Cases – The last chapter covers the emerging and future trends and how generative AI can be used in data engineering. It provides some key use cases where generative AI can be used in data engineering. Also, it throws light on serverless data processing and event driven architectures and how this is gaining prominence in data engineering. This chapter also touches upon data mesh and decentralized data ownership and how it promotes greater autonomy, scalability and data democratization. Finally, the book concludes with several real world case studies and how enterprises around the world are implementing data engineering use case in GCP and benefitting from it.