Why monitor record counts on Objects?
Objects with a significant number of records cause significant performance degradation to your Org due to data skew and Salesforce technical limits (you can read more about those limitations here). Keeping your data volumes under control is also one of Salesforce's Well-Architected principles.
Here are the key issues you might experience:
Slow Query Performance: Large data volumes increase the time it takes to perform queries and retrieve results, especially if the query isn't optimized for large datasets.
Search Delays: Indexing for large datasets can take longer, affecting the performance of search operations like free-form text searches.
Inefficient Reporting: Reports spanning large data volumes with multiple joins and filters experience long processing times.
API Performance Issues: Data extraction and loading via APIs can be inefficient, especially with large volumes of data, leading to timeouts or throttling.
Data Skew: Uneven distribution of data, such as a small number of parent records with many child records, can slow down the system and cause locking issues.
Sharing Rule Calculation Delays: Large data volumes can extend the time needed to calculate sharing rules, especially when adding or modifying records.
Degraded Performance from Soft Deletes: Deleted records stored in a recycle bin can still impact system performance until they are fully removed.
Storage Management Problems: Managing storage across multiple objects with large volumes of records can lead to slower overall system performance.
If your Org stores a lot of data in multiple objects, and you don't take proactive steps to address, the performance, usability, and eventually usefulness and adoption of your Org will suffer.
When to investigate record counts on Objects?
You should monitor Salesforce objects regularly and under specific conditions based on the guide. Elements can help you identify, segment, monitor, and take action on object that require intervention due to their object counts.
Here’s when to do so:
Regular Monitoring:
Ongoing Basis: Continuously monitor objects categorized as "Needs Monitoring" and "Needs remediation" (i.e., objects exceeding 10,000 or 50,000 records, or those within 10-15% of these thresholds).
Scheduled Reviews: Conduct periodic reviews (e.g., weekly, monthly) to track growth trends, update categorizations, and identify any emerging risks.
Threshold Alerts:
Near-Critical Limits: Set up alerts in Salesforce to notify you when an object is approaching critical thresholds (e.g., nearing 50,000 records). This ensures timely intervention before limits are breached.
After Major Business Changes:
New Processes or Integrations: Monitor closely after implementing new business processes, system integrations, or when launching major campaigns that might accelerate record creation.
System Upgrades: Review objects post-upgrade, as changes in system behavior could impact record growth or performance.
Before Peak Usage Periods:
High-Volume Events: Prior to expected high-volume periods (e.g., sales promotions, end-of-quarter reporting), ensure that all relevant objects are monitored to prevent performance bottlenecks.
Prerequisites
In order to follow this guide you need:
Salesforce Metadata Management license
Synced Salesforce Org into Metadata Dictionary
This solution guide also covers use-case for using MetaFields functionality. You can learn more about setting it up here.
MetaField definitions
In order to better analyze and segment your objects, you will need a custom attribute in your Metadata Dictionary, what we call a MetaField.
MetaFields to create
MetaFields to create
All of the MetaFields below should be enabled for following Metadata types only:
Custom object
Standard object
You will need to create following MetaFields to pro-actively categorize objects by your data storage policy:
How long data needs to be stored? (picklist)
Values:
"Short-Term (1 year)"
"Medium-Term (1-5 years)"
"Long-Term (5+ years)"
"Permanent"
Purpose: Allows you to classify the object based on the required data retention duration.
Data retention reason (picklist)
Values:
"Regulatory Requirement"
"Customer Contract"
"Business Process"
"Historical Reporting"
"Audit Trail"
"Data Analytics"
Purpose: Explains why the data needs to be retained for specified time period.
Archiving Strategy (picklist)
Values:
"Archive off platform"
"Migrate to Big Object"
"Delete"
"Keep on platform"
Purpose: Defines the strategy for dealing with the data once it's no longer actively used.
Data archiving condition (long text)
Purpose: Describes the conditions or events that can be used to decide which records to archive, migrate or delete. For instance "Cases for accounts that have stopped being customers over N months ago"
You will need to create following MetaFields to classify objects by actions to be taken when data storage becomes an issue:
Data storage risk (picklist)
Values:
Healthy
Needs monitoring
Needs remediation
Optimized
Purpose: Tracks the current classification of an object based on its record count and data storage utilization.
Remediation plan (multi-select picklist; conditional)
Values: (based on Salesforce's best practice guide)
Index Key Fields
Use Skinny Tables
Optimize Queries
Partition Data
Archive Old Data
Defer Sharing Calculations
Use Bulk API
Apply Batch Processing
Remove Unused Data
Reduce Data Skew
Purpose: Allows to choose remediation action.
Condition: The values should only be available if data storage is set to "Needs remediation" or 'Optimized'
Document your data archiving policies
Your Org has plenty of standard and custom objects. But most of the time there is no specific policy or consideration for how much data do we plan to store, for how long, and what to do with the data which is no longer needed.
Step 1: Document data policy when designing a new object
When you are designing a new custom object, or planning to install a new Managed Package, or even planning to purchase a new Salesforce Cloud (e.g. Industry Cloud, Revenue Cloud, etc.), we recommend that you review the objects that will be added to your data model with the business stakeholders and answer following questions:
How long data needs to be stored?
Why do we need to store the data for so long?
What is our archiving strategy for records we no longer need?
This will help you pro-actively plan and document your data strategy for the future.
You can create a proposed metadata component in your metadata dictionary for the planned objects and document the answer to those questions using rich text notes.
Limitation: MetaFields are currently not yet available on proposed metadata.
Step 2: Review and document data storage policy for existing objects
Step 2.1: Create custom view of metadata
Start by creating a custom view of metadata for custom and standard objects. Pick name, API name, metadata type, description to be included as columns. Also add the three MetaFields created earlier:
How long data needs to be stored? (picklist)
Data retention reason (picklist)
Archiving Strategy (picklist)
Data archiving condition (long text)
Step 2.2: Review and document object data storage policy with stakeholders
Sit down with business stakeholders (department heads and legal team) and review each object.
Tip: if you document stakeholders against your objects, then it is easier to identify who needs to be consulted on which objects.
Document the answers to the four MetaFields to ensure data storage policy is captured for all objects.
How long data needs to be stored?
Why do we need to store the data for so long?
What is our archiving strategy for records we no longer need?
How can we identify records that can be archived?
Example: It is common for case records to accumulate quickly over time. For a business that sells their product or services as annual subscription, up to 5 year deals. You may specify a condition that all case records need to be stored for up to 5 years. Or that they can be deleted within 3 months after the account churns and does not renew their contract.
Identify risky objects based on record counts
Step 1: Quick scan in Analytics 360
Watch this quick video from Brooke, our Salesforce expert, on how you can quickly identify objects with risky levels of record counts in your Org.
Step 2: Review and classify risky objects
Step 2.1: Create custom view of metadata
Start by creating a custom view of metadata for custom and standard objects. Pick name, API name, metadata type, description, record count to be included as columns. Also add the six MetaFields created earlier :
How long data needs to be stored? (picklist)
Data retention reason (picklist)
Archiving Strategy (picklist)
Data archiving condition (long text)
Data storage risk (picklist)
Remediation plan (picklist)
Step 2.2: Classify objects exceeding 10,000 records
Objects that have more than 10,000 records should be flagged as 'Needs remediation' and investigated because of potential data skew issue that can cause inefficiencies and slow down data processing.
Current limitation: currently you cannot filter a custom view of metadata by greater than / less than number of record counts. Scan the list visually to quickly find any objects with more than 10,000 records but less than 50,000 records.
You can also export the list of objects as a CSV file and filter it in a spreadsheet.
Just because an object has more than 10,000+ records does not mean that there is a data skew issue by default. You will need to perform investigation in your Org by creating custom Salesforce reports for the object records to identify any potential data skew and take action.
Issues to look for and how to remedy them
Issues to look for and how to remedy them
Ownership Skew: Objects with 10,000+ records owned by a single user can cause significant performance degradation, particularly during operations requiring record access changes, such as sharing recalculations or bulk updates.
This could lead to delays in critical business processes, affect user productivity, and disrupt workflows. Redistributing record ownership is not just a technical fix but a strategy to maintain smooth business operations and ensure that key processes are not hindered by technical bottlenecks.
Remediation plan value: Reduce data skew
What does it mean? : Redistribute ownership to mitigate this risk.
Lookup Skew: When 10,000+ records are linked to a single parent via a lookup relationship, this can lead to database locking and performance issues. These technical issues can slow down transactions and operations that are vital to business functions, such as order processing or customer service.
By evaluating and optimizing parent-child relationships, you can prevent these disruptions and ensure that the business processes reliant on these data relationships remain efficient and responsive.
Remediation plan value: Reduce Data Skew
What does it mean? : Optimize parent-child relationships to avoid these problems.
Account Data Skew: For Account objects with 10,000+ related records, there's a risk of account data skew, which can cause locking errors and slower processing. This could impact sales cycles, customer interactions, and other business activities tied to large accounts.
Assessing the need to split large accounts or optimize the data structure is crucial to maintaining business agility and preventing technical issues from affecting your organization’s ability to serve customers effectively.
Remediation plan value: Partition Data.
What does it mean? : Consider splitting large accounts or restructuring data to prevent issues.
Step 2.3: Classify objects exceeding 50,000 records
50,000 is a magic number in Salesforce, as a lot of queries are limited to that scale. Also, without properly indexed objects, beyond this point you will start experiencing some serious performance problems.
Classify all objects with more than 50,000 records as 'Needs remediation'.
Current limitation: currently you cannot filter a custom view of metadata by greater than / less than number of record counts. Scan the list visually to quickly find any objects with more than 50,000 records.
You can also export the list of objects as a CSV file and filter it in a spreadsheet.
Step 2.3.1: Identify objects that have a lot of records that can be archived
Earlier, you have documented data storage policies for each object in your Org. Use this information now to identify if any large-data-volume objects contain records that can potentially be archived.
Filter your custom view of metadata to only contain objects that for 'Archiving strategy' MetaField have value 'is not equal to "Keep on platform". This will show all objects where archiving strategy is set to delete records, archive them off platform, or store them in the big object.
The 'Data archiving condition' should contain criteria you will need to identify records that can be archived. But to identify the share of records that can be archived, and whether that takes you significantly below the 50,000 records threshold, you will need to run custom reports in Salesforce on the data to see how many records match those conditions.
Step 2.3.2: Identify actions to optimize performance on large-data-volume objects
Sometimes you cannot remove enough records to get below the 50,000 records threshold. Or even if you can, the object sees so much activity (e.g. case, task, contact) that you cannot always keep on top of staying within the limit. Therefore you need to take necessary actions to optimize queries, automations, and performance of metadata tied to those objects.
For objects that have more than 50,000 records, run the dependency grid. It will list any automations, like flows, apex classes, or apex triggers, that are tied to those objects. If there are no automations tied to those objects, then you are not facing any immediate challenges. But you should still consider taking action in order to prevent future challenges.
For objects with over 50,000 records and any flows, apex classes or triggers dependent on them, review them one by one to identify any potential issues.
Issues to look for and how to remedy them
Issues to look for and how to remedy them
SOQL Query Limits in Apex Classes: SOQL queries are limited to 50,000 records in Apex code. Exceeding this limit can cause failures or prevent a query from returning all relevant data.
Remediation plan: Optimize queries
API Call Limits: Large datasets can strain API call limits, particularly during integrations or bulk operations.
Mitigation: Use Bulk API (to get data from the object in integrations) or Apply Batch Processing (for internal automations)
Trigger Execution Limits: Salesforce triggers process up to 200 records at a time. Large datasets may hit governor limits during bulk updates or complex transactions.
Mitigation: Review and optimize triggers to prevent these issues.(Note: code quality factors into this significantly– sub optimal code can result in a need to throttle back from 200)
Indexing and Selectivity: Non-selective queries can lead to full table scans, significantly impacting performance.
Mitigation: Index Key Fields and Optimize Queries
Step 3: Audit sources of data creation
Should this object really have that much data?
Should the number of records be accumulating that quickly?
These are the sorts of questions that are not often asked, but are critical to monitoring and securing performance of your Org.
Here is what you need to do:
Assess whether records are being created intentionally as part of a controlled process, and identify the systems or processes contributing to record creation.
Identify anomalies in automated processes related to this object, and determine if automation should be adjusted to better manage record creation and lifecycle.
In the right panel for the selected object, in the usage tab, you can see the list of automations tied to an object and even identify apex classes or flows responsible for creating new data.
Review those automations and decide whether they are still needed, or are they legacy automation that support a long forgotten or deprecated business process.
Step 4: Identify and resolve unusual data patterns
You should also investigate unusual spikes or drops in record creation, aligning them with business events or changes in process. Anomalies in data patterns can often reflect changes in business activities, such as a new marketing campaign, changes in customer behavior, or operational shifts.
Understanding these links helps you interpret record counts within the broader context of your business strategy. And this helps you identify data spikes or drops that cannot be explained by expected business activity - and hence point to suspicious activity that should be investigated.
For the selected object, right-click / use context menu and select 'Open adoption insights'. This will open a dashboard for the object with record population data for the selected time period. You can use the Record count over time chart to identify any suspicious data patterns.
Take action
After the review is finished, you will end up with a list of objects classified by data storage risk as 'Needs remediation'.
Custom views of metadata come equipped with many single and bulk operations. You can raise user stories and document tasks to index key fields, optimize queries, or any other identified remediation actions.
We propose that you filter your object list by remediation plan (e.g. remediation plan is 'Archive old data') and then you can bulk-create stories for all objects with the same remediation plan. You will end up with multiple stories, each linked to a different object, specifying the needed remediation action.
You can then pick up those stories from your backlog and deliver then when there is capacity.
