This chapter covers two important aspects of developing an enterprise-level application on the Force.com platform. While asynchronous processing and Big Data volumes are not necessarily linked, the greater governor limits and processing power of the platform are mostly to be found in the async processing mode. However, making sure that your interactive user experience is still responsive when querying large volumes of data is also very important.
In this chapter, we will first review how to ensure SOQL queries are as performant as possible by understanding how to profile queries and make use of the standard and custom indexes. This will benefit both interactive and batch (or async) processes in your application as well as how they leverage native platform features such as reporting.
We will then take a look at the options, best practices, design, and usability considerations to move your application processing into the async mode, once again leveraging the Service layer as the shared backbone of your application, before finally understanding some thoughts and considerations when it comes to volume testing your application. We will cover the following topics in this chapter:
- Creating test data for volume testing
- Indexes, being selective, and query optimization
- Asynchronous execution contexts
- Volume testing
Throughout this chapter, we are going to use the RaceData__c object. We will be extending it with additional fields and running a script to populate it with some test volume data, before exploring the different ways of accessing it and processing the data held within it.
In the Formula1 racing world, a huge amount of data is captured from the drivers and the cars during the race. This information is used to perform real-time or post race analysis in order to lead to further improvements or troubleshoot issues. Because of the huge volumes of data involved, the FormulaForce application considers ways to summarize this data such that it can be removed and later reloaded easily if required.
The following functionalities will be added to the FormulaForce application to process this information and help demonstrate the Force.com features described in this chapter. The Apex classes and Visualforce page for this functionality is included in the sample code for this chapter.
- A Visualforce page able to perform some What-If analysis on the race data
- A Batch Apex process to validate and associate race data with contestants
The sample code for this chapter also contains updates to the Race Data object, as follows:
- The Type field represents the type of race data, for example, Oil Pressure, Engine Temperature, Sector Time, Fuel Level, Pit Stop Times, and so on.
- The Value field is a numeric value dependent on the Type field.
- The Lap and Sector fields are numeric fields that represent the lap and sector number the data was captured on.
- The Year, Race Name, and Driver Id fields are text fields whose values, when later combined together, can be used to form a key to link each Race Data record with the appropriate Contestant record (which has its own relationships with the Driver, Race, and Season objects). In this use case, we assume that the electronic systems generating this raw data do not know or want to understand Salesforce IDs, so the FormulaForce application must correctly resolve the relationship with the Contestant record.
- The Contestant lookup field references the applicable Contestant record. As the incoming raw data does not contain Salesforce record IDs, this field is not initially populated. The FormulaForce applications logic will populate this later, as will be shown later in this chapter.
- A new Race Data ID external ID field has been added to the Contestant object. The Apex code in the
ContestantsDomain class has been updated to ensure that this field is automatically populated as Contestants are added and that its value can be used during the processing of new Race Data to correctly associate Race Data records with Contestant records.
The Race Data object uses an Auto Number field type for its Name field. This ensures the insert order for the data is also retained.
Tip
Note that we have added the Race Data ID field to an object already released in previous versions of the package, while the Contestants Domain class has been updated to populate this field for new records. The records already existing in subscriber orgs will not contain this value. Force.com provides a means to execute the Apex code after installation (and uninstallation) of the package, such that you can include the code to trigger a Batch Apex job to update this field value with the appropriate value on existing records. Refer to the System.InstallHandler interface in the Apex Developers Guide.
Just like me, I doubt if you have a real Formula1 race team or car to provide raw data input. We will be using an Apex script to generate the test data, though in reality race data may be inserted through a CSV file import or through an integration with the car itself via the Salesforce APIs.
This test script will generate 10,000 rows, which is about enough to explore the platform features described in this chapter and the behavior, Force.com SOQL profiler, which makes different choices dependent on row count.
Due to the volumes of data being created (and deleted if you rerun the scripts), you must run each of the following Apex statements one by one to stay within the governors via the Execute Anonymous prompt from your chosen IDE or developer console:
- Run
TestData.resetTestSeason(); - Run
TestData.purgeVolumeData(false); - Run
TestData.purgeVolumeData(true); - Run
TestData.createVolumeData(2017, 'Spain', 52, 4);
Note
Step 1 of the preceding steps will run a script that will clear down all other data from other FormulaForce objects in order to set up test Season, Race, and Driver data for the rest of the chapter. You can repeat Step 4 to generate a further 10,000 Race Data records for additional races—something we will do later in this chapter. This step might take a few minutes to complete.
Once the script has been executed, go to the Race Data tab and review the data. The values shown in the Value column are random values, so your results will differ.
You should also see the following row count if you review the Storage Usage page:
