data architecture design is about setting up a plan for how an organization gathers, stores, and uses its data. in this chapter, we'll look at different methods and ideas in data architecture design to help you understand how to create a system that meets your organization's data needs effectively.

table of content

difference between oltp and olap
operational and analytical data
symmetric multiprocessing and massively parallel processing
lambda architecture
kappa architecture
polyglot persistence and polyglot data stores

difference between oltp and olap

online transaction processing (oltp) and online analytical processing (olap) are two main types of data processing systems.

online transaction processing (oltp) systems are designed to quickly handle real-time transactions using relational databases. they allow users to easily create, read, update, and delete data. these systems can support many users at once and are commonly used in places like cash registers and online banking.

features of online transaction processing(oltp)

in this section, we will look at the main features of oltp systems that are important for handling daily transactions. these include.

focuses on fast data processing.
handles many small transactions.
keeps data accurate for multiple users.
usually uses organized databases for efficiency.

example: a bank's system for processing customer withdrawals and deposits.

online analytical processing (olap) systems are designed for data analysis and reporting. they handle complex queries, allowing users to quickly view data from different angles. olap databases often use pre-summarized data in structures called cubes, making it easy to find summarized information.

features of online analytical processing(olap)

in this section, we'll look at the main features of olap systems that make them important for analyzing data and creating reports. these include.

focuses on retrieving data and doing complex calculations.
handles fewer, but more complicated queries.
often uses simpler databases for quicker analysis.
supports viewing data in multiple ways.

example: a retail company's system that analyzes sales trends over different regions and time periods

key differences between oltp and olap

this section shows the main differences between oltp and olap systems. knowing these differences helps you choose the right system for handling transactions or analyzing data. the table below shows the main features.

feature	oltp (online transaction processing)	olap (online analytical processing)
processing type:	handles everyday transactions	focuses on analyzing data
data type:	works with current operational data	uses summarized, consolidated data
purpose:	supports daily business activities	aids in making decisions
transaction frequency:	processes many transactions frequently	works with data occasionally
query complexity:	handles simple queries quickly	manages complex queries for deeper insights
response time:	provides instant results	may take seconds to hours
database size:	typically in gigabytes	often in terabytes

operational and analytical data

operational data is real-time data used for managing daily tasks. it is processed by oltp systems and gives a current view of the business. operational data is typically high in volume, meaning it consists of large amounts of information generated frequently, which helps in making quick decisions.

features of operational data

when we talk about operational data, we're looking at the information that supports daily business tasks. its features include.

used for daily business activities
always changing and getting updated
usually stored in oltp systems

example: the amount of items currently in a warehouse.

analytical data is created by transforming operational data to give a historical view. it is managed by olap systems and data warehouses. this data helps us understand trends and patterns over time, making it useful for reports and machine learning. typically, analytical data has less information and is often summarized from larger sets that are processed in batches.

features of analytical data

when looking at analytical data, it's important to know its key features, as they are important for clear analysis and decision-making. these features include

used for making business decisions
mostly contains older data that doesn't change
usually stored in olap systems or data warehouses

example: sales data from the past five years used to predict future trends.

operational data is used to monitor daily activities, while analytical data helps with long-term decisions. both types of data are important for running a business effectively. oltp systems manage operational data, and olap systems or data warehouses are used for analytical data.

symmetric multiprocessing and massively parallel processing

symmetric multiprocessing (smp) is an older database design where multiple processors use the same server's memory and storage. it works well for daily transactions (oltp) but has difficulty handling large amounts of data found in data warehouses. you can improve performance by adding more processors to the same server.

features of symmetric multiprocessing

this section looks at the key features of symmetric multiprocessing (smp), where multiple processors share the same memory. these features include.

all processors use the same memory.
good for systems that need some parallel processing.
easier to manage and program.
limited scalability, usually up to 32 processors.

example: a database server that uses multiple processors to handle queries at the same time.

massively parallel processing (mpp) is a newer design that uses multiple servers, each with its own memory and storage. it allows you to increase capacity by adding more servers. in mpp, data is spread across these servers, and tasks are divided so they can be processed simultaneously, making it more efficient for large datasets.

features of massive parallel processing

this section covers the key features of massive parallel processing (mpp), where each processor has its own memory. these features include.

each processor has its own memory and operating system.
highly scalable; can use hundreds or thousands of processors.
great for handling very large datasets.
more complex to manage and program.

example: a big data warehouse system that handles complex queries across multiple servers.

analogy: think of searching for a card. if one person is looking (smp), it takes longer. but if several people search for fewer cards each (mpp), it's much faster.

both smp and mpp systems originally started as local installations, but now many cloud-based options are available.

lambda architecture

lambda architecture is designed to handle large amounts of data using both batch processing (for historical data) and real-time stream processing (for live data). it combines these methods to give a complete view of information.

key concept of lambda architecture

this section explains the main ideas of lambda architecture for processing data. these concepts are:

batch layer: processes large amounts of historical data.
speed layer: processes real-time data streams.
serving layer: responds to queries using both batch and real-time views.

key principle of lambda architecture

here, we'll look at the main principles of lambda architecture that make it work well. these principles include.

dual data model: uses one system for batch data and another for real-time data.
unified view: shows both batch and real-time results in one place.
separate processing layers: batch and real-time processes work independently for easier development and scaling.

lambda architecture process

in this section, we'll look at how the lambda architecture process works for managing data. these steps include.

data layer: collects data from different sources, whether it's coming in continuously (streaming) or at set times (periodic).
stream layer: quickly processes the latest data, trading some accuracy for faster results. this data is often stored in a data lake for easy access.
batch layer: processes all the data together to ensure accuracy, serving as the main source of information.
presentation layer: decides when to use data from either the batch or stream layers based on what users need.

advantages of lambda architecture

here are the main advantages of lambda architecture for data management.

it processes data efficiently, giving both real-time and historical information.
it gives a clear understanding of data, which is important for modern applications.

use cases and limitations of lambda architecture

lambda architecture works well for applications that need both real-time and historical data, like recommendation systems. however, it can be complex and may not be the best choice for.

high real-time data: lambda architecture might not work well for systems that need to process a lot of real-time data quickly, where kappa architecture could be a better choice.
state tracking: if you need to monitor events over time, lambda architecture might not be the best option because it doesn't keep track of the state.

kappa architecture

kappa architecture is all about processing data in real time. it doesn't handle batch data like lambda architecture does. it is built to manage high volumes of data with quick responsiveness.

key features of kappa architecture

this section covers the main features of kappa architecture that make it effective. these features include.

real-time processing: data is processed as soon as it arrives, allowing for faster responses.
single event stream: all data flows through one main stream, which makes it easy to scale and recover from failures.
stateless processing: each piece of data is handled separately, so there's no need to remember past data. this makes it easier to expand the system.

advantages of kappa architecture

in this section, we'll discuss some key advantages of kappa architecture that make it a great option. these include.

easier to build and maintain than lambda architecture.
offers consistent processing for both real-time and historical data.
simpler to understand and debug.

limitations of kappa architecture

here, we will look at some key limitations of kappa architecture, which can affect its use. these include.

complexity: it can be difficult to set up and maintain due to its structure.
no batch processing: kappa struggles with old data and bulk processing, making it less suitable for analyzing historical information.
limited ad-hoc queries: it may not work well for quick searches that require a lot of historical data.

when to use kappa architecture

kappa architecture is best for applications that need fast, real-time data processing without analyzing historical data. it works well in these situations.

real-time applications: it's perfect for services that need quick updates, like streaming or stock trading.
focus on current data: this architecture is perfect for situations where you only need the latest information.
for historical data needs: if you also want to analyze past data, it's better to choose lambda architecture instead.

polyglot persistence and polyglot data stores

polyglot persistence means using different data storage technologies within a single application, depending on the type of data and how it will be used. it's about selecting the best tool for each specific task, similar to how polyglot programming uses multiple programming languages to utilize their strengths.

on the other hand, polyglot data stores involve using different data storage solutions across an organization. each type of data store is optimized for particular data types or applications, allowing teams to choose the best option rather than relying on a single solution for everything.

for example, in an e-commerce platform, instead of storing all data in one database, you might use.

key-value store for fast retrieval of shopping cart and session data.
document store for easy management of completed orders.
relational database for structured inventory and pricing data.
graph store for customer relationships.

while using multiple data storage technologies can increase complexity due to the need to learn different systems, the benefits include improved speed and performance. choosing the right storage for each data type leads to better application performance and development speed.

advantages of polyglot persistence

here, we will see some key advantages of polyglot persistence, which allows the use of different databases for different needs. these include.

flexibility: you can use different types of databases based on what you need.
efficiency: it helps your system run better by picking the best database for each kind of data.
scalability: as your data grows, it easily supports new systems and technologies.

advantages of polyglot data stores

this section looks at the main advantages of polyglot data stores, which combine different storage solutions for better data management. these advantages include.

diverse capabilities: it combines different storage solutions to handle all kinds of data effectively.
better data management: makes it simpler to manage and analyze data from various sources.
cost-effective: saves money by using the right database for each task.

understanding the different ways to design data architecture is important for making efficient data systems. the best method depends on your organization's needs and the type of data. successful architectures commonly combine different methods to manage data well.

Save or share this article