Throughout this chapter, we've covered a lot of ground with regard to contacts. However, our emphasis has been on understanding the types of data collected, navigating through the system, and working with search tools and results. Let's wrap this chapter up by looking at alternate workflows and record maintenance through deduping tools.
We are working within a CRM, a Constituent, or Contact relationship management tool. So it's natural to view your data through the lens of contacts. Just about everything we've covered thus far has done this; we've either been working inside a single contact record or we've conducted a search to take action on a group of contact records.
However, sometimes it may seem more natural to start with the action instead of the contact record. CiviCRM provides several options for this type of alternate workflow.
On the left sidebar, you will see a Create New button and a drop-down menu. This button is always visible in Joomla!, and is a default-enabled block in Drupal. Drupal users may configure it through the Drupal Administer | Site building | Blocks tool. In addition to options for creating contacts, there are options available for creating an activity, contribution, e-mail, event registration, membership, pledge, group, or tag.
Many of these options are also listed under the appropriate main menu item (for example, New Contribution is listed under the Contributions menu).
Choices that will create records attached to a contact will have a contact field on the form where you can begin typing an existing contact name or choose to create a new contact from the drop-down.
Creating a new record "on-the-fly" causes a pop up window to appear with a few contact fields. This form is configurable through Administer | Customize | CiviCRM Profile. By default, it contains only a few required fields. You may want to expand what fields are included in order to ensure that the system users collect more information when they are entering records into the system.
Note
The profiles used for "on-the-fly" contact creation are entitled New Household, New Individual, and New Organization. Notice in the CiviCRM profile listing page, that these are marked as Reserved. Throughout CiviCRM, any records marked as Reserved cannot be deleted, and will be used for special purposes in the system. It is recommended that you do not make any radical modifications to reserved records, as it may have unintended consequences in the system. For example, we advise against renaming the New Individual profile, and then using it as a contact signup form.
Data is king. Even the most advanced CRM tools will be virtually useless if the data it is handling is incorrect, messy, or outdated. Your system will live or die based on the principle of garbage in/garbage out.
What steps can you take to improve the quality and reliability of your data? One important way is to reduce record duplication.
Record duplication is an unavoidable reality, especially once you begin to expose forms to the front of your website and invite constituents and site visitors to submit their own data. Fortunately, CiviCRM has several ways of dealing with record duplication.
There are three strategies we will use in the fight against duplicates:
- Policies and procedures: Yes, the first tactic is a human resource and management strategy. Anyone who is working in CiviCRM as an administrator must understand the importance of reducing duplication. You can start this is by enforcing a policy where every time a user is going to work on a contact record, they begin by doing a search for the contact. Simple, right? How many times do people enter the system and immediately begin by creating a new record? CiviCRM's auto-fill search tools will help significantly, but they will only benefit you if they are used.
- Inline deduping: While creating or editing a contact record, CiviCRM does a lookup to see if there is a potential match with an existing record. If so, you are given the option of visiting the potential duplicate record or continuing to save your current record. CiviCRM uses deduping rules (discussed in the following point) to define what potentially constitutes a match. The default fuzzy rules are used for inline deduping. You may also trigger a dedupe check with the Check for matching contacts button.
Understand that this functionality is a safety net—don't rely on it as your first line of defense. One potential loophole with this tool appears if the new contact has minimal data completed and consequently there is insufficient information to verify a duplicate match. For example, the default individual rule matches on first name, last name, and e-mail. If I add a record with the first name, last name, and address, it may not flag a potential duplicate because the e-mail didn't match.
- Find and Merge Duplicate Contacts: Available through the Contacts menu, this tool can be used to search your database for potential duplicates and merge them.
Regardless of how well you adhere to the first two policies, you will need to schedule periodic maintenance routines, where you manually run the Find and merge duplicate tool on your records. Depending on the size of your organization and where you are regularly obtaining new data (for example, through online forms), you may schedule regular deduping on a weekly, monthly, or semi-annual basis.
When you first click on Contacts | Find and Merge Duplicate Contacts, you will see a list of six matching rules—two for each primary contact type, designated as either Fuzzy or Strict, and flagged as Default. The Fuzzy or Strict designation is used for your reference, and also impacts how the default rules will be used by the system.
The default Fuzzy rule is used for the automated inline deduplication check we just described. When a contact record is entered, the appropriate default fuzzy rule (based on the contact type) is used to determine if there's an existing match. It may be fuzzy/looser in its matching criteria because the user is directly involved in the review and acceptance/rejection of the duplicate is possible.
The default Strict rule is used for automated deduplication checking for frontend forms and bulk imports. For example, if you have a newsletter signup form using a profile and have chosen the option to update the record on a duplicate match, the strict default rule will be used to determine if the form submission matches an existing record.
Note that it is your responsibility to actually configure the rule settings for the level of strictness and fuzziness you desire. In most cases, you will find the default settings adequate for these purposes.
From this screen, you initiate a manual deduplication check, and can edit or create new rules for a contact type.
Note
FPAGM recently hosted their annual conference, drawing over 350 people from the state to learn more about food pantry management best practices and hear about other homeless action initiatives happening throughout the country. A majority of the participants registered online. While this significantly eased the amount of data entry work required by staff, it did generate duplicate records in the system as many participants were not careful while filling out the forms.
A majority of the Association's existing contact records (prior to the conference) did not have an e-mail address, so the default matching rules were too strict for their needs. They decided to create a new fuzzy individual rule to match on first name and last name.
The matching rules configuration form consists of three columns that may be configured: Field, Length, and Weight.
- Field: This indicates the data to be tested as a potential match.
- Length: This indicates how many characters in the field value should be measured. A lower value will result in more matches, as more records are likely to find a match if a shorter value is used. For example, a length of four in the last name field would match Johnson and Johnston, whereas a length of six would not match (Johnso != Johnst). In the preceding example, we are seeking to have a fairly fuzzy match, so we've defined a short length. Leaving the field blank will require a match on the entire field value.
- Weight: This is used to calculate the likelihood of a match. Using the preceding rule, a match on both first name (10) and last name (10) yields a total score of 20, which meets the threshold value at the bottom of that column. The example above is very basic; we are simply requiring a match on all fields for the system to consider the contact a potential match. Consider an alternate configuration:
- First name (10), Last name (10), E-mail (5), City (5), Total threshold = 25
- In this configuration, a match on first name, last name, and e-mail, OR first name, last name, and city, would both meet the threshold and return a potential match
You will want to think carefully through your settings for dedupe rules. Rules that are too strict may result in missing some valid duplicates, and rules that are too fuzzy will yield long result lists that are cumbersome to work with. In particular, review the default Strict rules, as these impact deduplication/merging in hands-off environments (online forms and imports).
Once you've configured your rule, click on Save to return to the rule list and click to use the rule. By default, a rule will be run against the entire database (with the contact types appropriate to the rule). However, you can also choose to restrict the process to a group. This is helpful if you have a very large database and want to process deduplication in smaller chunks of targeted records.
After triggering (running) a rule, CiviCRM provides a list of potential duplicate contacts. The two potential matches are listed, along with the total weight (higher weighted matches are first), and a link to merge the records.
The merge function is where the real power of this tool is realized. Both records are presented in a grid, and you decide which fields will be moved from the duplicate on the left to the main contact on the right. In this way, you can choose exactly what fields will be merged. The tool will also combine any related records, such as activities, contributions, membership, and so on. This is important, as you would not want the historical data contained in those records lost as a result of the merging process.
Note
Be careful! Dedupe/merge is a powerful tool. Once you choose to merge the records, the duplicate contact will be deleted from the system. If the contact trash is enabled, the duplicate will be moved to the trash.
As you can see from the preceding screenshot, we've chosen to merge the e-mail (missing in the main contact), and the related contribution, participant, and activity records (in order to preserve the historical data).
After completing the merge, you are returned to the resulting merged contact. If you plan to merge multiple records found after running the rule, use the middle mouse button or Ctrl + click to open the merge tool in a new browser tab/window.