Building for Large-Scale Database Operations

In the area of enterprise software development, developers have always built applications that deal with large amounts of data. In the early days of computing, systems used to span rooms bigger than the ones we currently live in, and data was stored in a flat file format, whereas today, systems have shrunk so much that in the same sized room that used to store a single system, we can now have thousands of systems running, each coordinating with the others, providing us with machines that can crunch data at the speed of light. Over time, the way data is stored has also evolved from using flat files to sophisticated database management systems.

With enterprises growing in size and their ever-expanding operations due to emerging fields, the amount data that needs to be processed by enterprise applications is also growing, and this makes it important to understand how to build our applications to deal with large-scale database-related operations. Though building for large-scale database operations can never be a one-approach-fits-all solution, we'll cover some points that are common to building applications that can scale easily to handle the increase in data, the requirements of schema modification, increasing application complexity, and so on.

Although there are multiple types of databases, such as SQL, NoSQL, and Graph, that can be used to store application data, depending what kind of application is required by the enterprise, this chapter focuses on the use of relational database management systems using SQL, due to their vast popularity and their ability to handle a large amount of use cases.

By the end of this chapter, you will have learned about the following:

• Using Object Relational Mappers (ORMs) and the benefits they provide
• Structuring database models for efficiency and ease of modification
• Focusing on maintaining database consistency

• The differences between eager loading and lazy loading
• Taking advantage of caching to speed up queries