Starting with the storage block, data is stored in Data Cloud in the Parquet file format in Amazon S3. Parquet is a columnar storage that provides warehouse-type columnar access. There are three layers of storage: cold, warm, and hot. For hot storage, it uses Amazon DynamoDB. Additionally, it has an SQL metadata store and cached metadata security (see Figure 4.2).
Figure 4.2: Storage layering in Data Cloud
On top of the storage layers sits the big data processing layer. Data processing occurs using Apache Spark and Presto. Airflow is used for data orchestration. Data orchestration is the process of retrieving siloed data from different sources, combining, organizing, and making it available for analysis.
Data Transformation, Lighting-enabled Data Tables, and Model Mappings
On top of the big data processing layer sits the data transformation layer. Salesforce has a table abstraction feature called Iceberg. In Salesforce, an iceberg is a term used to describe a complex data model where most of the data is hidden or not easily visible on the surface. It refers to the idea that, like an iceberg, the majority of the data is below the surface and not immediately visible.
In Salesforce Data Cloud, data transformation, lightning-enabled data tables, and model mappings are all related to the process of data management and analysis. Here is an overview of each feature:
- Data Transformation: Data transformation refers to the process of converting data from one format to another. In Data Cloud, data transformation is performed using a data mapping tool that allows users to map source data fields to target data fields.
With data transformation, users can perform various data operations, such as filtering, aggregation, and enrichment. This allows them to standardize and normalize data, making it easier to analyze and use for personalized experiences.
- Lightning-enabled Data Tables: Lightning-enabled data tables are a feature of Salesforce that allows users to view and interact with large datasets. These tables provide a user-friendly interface for browsing, searching, and filtering data.
In Salesforce Data Cloud, lightning-enabled data tables can be used to analyze customer data and create personalized experiences. Users can apply filters, group data, and perform calculations, allowing them to identify trends and insights that can inform marketing and engagement strategies.
- Model Mappings: Model mappings are a way of connecting data sources and targets in Salesforce Data Cloud. They provide a standardized way of defining how data should be transformed and delivered to different systems.
With model mappings, users can define data mappings between different systems and data models, such as Salesforce objects, external databases, or marketing automation platforms. This allows them to maintain a consistent view of customer data across different systems, improving data quality and reducing errors.
For data flow architecture in Salesforce Data Cloud, please refer to Figure 4.3.
