Cross-System Identity Resolution for Human Services Organization

WHAT IS IDENTITY RESOLUTION?

Identity resolution is the process of determining that records in different systems refer to the same person, in the absence of a shared key. It is one of the most common problems in enterprise data and one of the most frequently underestimated.

The work divides into two families of technique. Deterministic matching applies explicit rules — same Social Security number, or same name and same date of birth — and produces a link only when a rule is satisfied exactly. Probabilistic matching scores the similarity of records across many fields at once and links them above a confidence threshold. Deterministic matching is auditable and explainable; you can always say precisely why two records were joined. Probabilistic matching catches more true matches, at the cost of being harder to justify to anyone who asks about a specific pair.

The output of either approach is typically a master person index: a central table holding the assigned universal identifier, the corresponding identifier in each source system, the standardized fields used for matching, and flags for records that need human judgment. The source systems themselves are left untouched. Corrections and harmonization live in the warehouse, so no program team loses control of its own data, and no matching error can propagate back into a clinical or housing record.

One property of this process is worth stating because it surprises people. Matching is transitive: if record A links to record B and B links to C, all three become one person. That is usually correct, and it means a single bad link can merge two people who are not the same. The strictness of the rules is therefore not a matter of taste.

POINTS OF NOTE REGARDING SOLUTION

The Rule Hierarchy, and What Each Rule Actually Costs

We used a ranked cascade of deterministic rules rather than a single probabilistic score. Each pair of systems is compared with the same rules in strength order, and a record links on the first rule that finds a partner. The strongest is a shared Social Security number. Below that sit name plus date of birth, name plus birth year within one year, name plus phone, name plus email, and name plus address. A final rule handles a specific known failure: identical date of birth with an SSN differing in exactly one digit, which is a data entry error rather than a different person.

Two design decisions in that cascade matter more than the rest.

The first is that name-only agreement is computed and reported but never links records. Two people sharing a name is common, and in a population of tens of thousands it is close to guaranteed. Those cases surface in the report flagged for manual review rather than being resolved automatically.

The second is that we validated the weaker rules against the strong one instead of assuming they were sound. On the pairs where both records carried an SSN and therefore matched on rule 1, we checked how often the weaker signals would have agreed: full names matched about 91 percent of the time, dates of birth about 96 percent, and birth year within one year about 99 percent. That is the evidence for allowing name and date of birth to stand in for SSN when SSN is missing.

The same check exposed the cost. Among pairs found by the name-and-date-of-birth rule where both records happened to carry an SSN, roughly 95 disagreed on that SSN. Some are typos and some are genuinely two different people who share a name and a birthday. We documented that error rate in the deliverable rather than leaving it implicit, because a matching system that reports no error rate is not more accurate — it is only less honest. Anyone using these numbers for funder reporting needs to know the linkage is very good and not perfect.

Minimizing Identifier Exposure in the Warehouse

Matching people requires the most sensitive fields an organization holds. Reporting on them does not, and we treated those as separate questions rather than one.

Social Security numbers and full dates of birth are used inside the matching stage and are not written to any gold table. The reporting layer carries only whether an SSN is on file and the person’s birth year. The Universal ID is also never written back to the source systems; it exists in the warehouse alone. That constraint costs some convenience — it means the operational systems cannot display the cross-system link — and it means an identity resolution process still under refinement cannot contaminate a clinical or housing record of record.

Standardization also does more privacy work than it appears to. Shared contact values were a real problem: a phone number or email address appearing on more than five people within one system is an office line or a program inbox, not a personal contact, and matching on it would merge unrelated people wholesale. Blanking those values improved accuracy and removed a category of spurious linkage at the same time.

CONCLUSION

The organization now has a single defensible participant count, a view of who appears in which systems, an explicit list of where systems disagree on a person’s name, birth year, or contact details, and a queue of cases a human needs to adjudicate. Records that cannot be matched by any rule remain visible as single-system participants rather than disappearing.

Two limitations belong in any honest account of this work. The Universal ID is stable between runs but not yet permanent — when a record from a new system links into an existing group for the first time, that group’s identifier can change. It will be frozen once matching stops being refined, and until then it should not be treated as a durable external key. Separately, one source contributed a one-time snapshot rather than a refreshing pipeline, so that system’s data reflects a fixed date rather than current state.

We were also asked whether a commercial master data management platform should be purchased for this. Our recommendation was to defer. Buying a platform before the matching rules, data quality issues, and stewardship workflows are understood tends to encode confusion into expensive software. Building identity resolution natively first means the organization learns from its own real conflicts, and if a platform is later adopted, it can be configured against requirements that are actually known.

BESPOKE SOLUTIONS FOR YOUR ORGANIZATION

Boxplot is passionate about working with all clients, regardless of their previous level of experience in data. If your organization is looking for a solution similar to the one described in this article —or any other data-oriented capability— let us know by contacting us here.