You imported 40,000 contacts. Your duplicate rules are on. You’re looking at 6,000 dupes. You set up duplicate detection rules before the migration. You checked them. They work in the UI: create a contact with a matching email address, and the warning fires as expected.
Then you run the import. Or the integration sync kicks off. Or Dual-Write pushes vendor records from Finance & Operations. And you’re staring at thousands of duplicates that your rules should have caught. Nothing is misconfigured. This is how it works by default.
Why duplicate detection doesn’t fire on Web API calls
When a record is created or updated through the Dynamics 365 Web API, duplicate detection does not run unless the request explicitly opts in.
The mechanism is a request header:
MSCRM.SuppressDuplicateDetection: false
Setting this header to “false” tells Dataverse to run duplicate detection on the call. The default behavior, when the header is absent, is equivalent to setting it to “true”: detection is suppressed.
Most integrations, migration tools, custom connectors, and platform features don’t set this header. That includes Dual-Write, KingswaySoft imports, Power Automate’s Dataverse connector (depending on how the action is configured), and most custom Web API clients built in-house. Unless the developer explicitly added the header, duplicates are silently coming in.
This is documented behavior. It’s not a bug. Microsoft’s position is that suppressing detection by default improves API performance. That’s a reasonable call for a general-purpose API. For a CRM migration, it’s the gap that bites you two weeks before go-live.
Where this shows up on real projects
Dual-Write sync. Dual-Write pushes records between Dynamics D365 CE and Finance & Operations via a continuous sync. The Web API calls it generates don’t set SuppressDuplicateDetection: false. The result is a well-documented pattern in the community: duplicate vendor and customer records that accumulate silently, even when detection rules are configured correctly.
One D365 consultant described it directly: “I set up duplicate detection rules, but it does not help.” That’s not a misconfiguration. That’s the default behavior.
KingswaySoft and SSIS imports. Bulk migration tooling typically writes records via the Web API in batches. Unless the connector is explicitly configured to pass the detection header, which requires knowing to look for it, duplicates from the source system land untouched.
Power Automate Dataverse connector. The Add a new row and Update a row actions in Power Automate don’t surface duplicate detection as an obvious option. Records created through automated flows bypass detection unless the flow is explicitly built to handle them.
Custom Web API clients. Any in-house integration, a custom sync from legacy ERP, a marketing automation connector, or a third-party data feed, inherits the same default. If the developer didn’t know about the header, detection isn’t running.
CreateMultiple and UpdateMultiple. These newer message types, introduced to improve bulk write performance, follow the same suppression default. If you’re using them for high-volume imports, the performance gains come with the same duplicate detection gap.
The fixes you can apply today, and where each one runs out
Set the header in the integration layer
If you control the integration or migration tool, adding MSCRM.SuppressDuplicateDetection: false to each Web API request is the most direct fix. For KingswaySoft, this is a connector-level setting. For custom clients, it’s a one-line header addition.
Where it runs out: You need to control and configure every integration individually. On a project with five data sources, that’s five places to get it right. On the next project, you start from scratch. And it only works for entry points you know about at project time: the integration nobody scoped for shows up after go-live.
It’s also worth noting what this header doesn’t do. When SuppressDuplicateDetection is set to false, a detected duplicate results in an API error: the incoming record is rejected outright. If your integration is pushing updated information onto existing records (a contact’s new phone number, a refreshed account address), this mechanism discards that data rather than merging it. For use cases involving record enrichment alongside deduplication, the header flag is the wrong tool entirely.
Write a plugin on the Create/Update message
A server-side plugin registered on the Create or Update message can enforce duplicate detection logic regardless of how the record was created. This gives you coverage that isn’t dependent on the calling client setting the right header.
Where it runs out: Plugins are per-entity. A plugin written for Contact doesn’t cover Account, Lead, or any custom table. On a typical CE implementation with custom entities, you’re writing and maintaining multiple plugins. They also need to be deployed and managed across client environments, and when the client upgrades or customizes, the plugin becomes something you own.
Post-load cleanup scripts
Run a bulk duplicate detection job after the migration completes. Catch what got through, merge manually or via script.
Where it runs out: The bulk duplicate detection job in Dataverse caps at 5,000 records. On any mid-market migration, that ceiling is hit before you’ve touched most of the dataset. You’re running the job in batches, tracking what’s been processed, and hoping nothing new came in between runs. This is a cleanup operation, not a prevention strategy, and it does nothing about duplicates that arrive after go-live through ongoing integrations.
Native duplicate detection rules
The rules work well for what they were built for: catching duplicates entered through the Dynamics 365 UI. They fire reliably on form saves. They also have a user bypass: the Ignore and save option lets any user dismiss the warning and save the record anyway. There’s no enforcement mechanism.
Where it runs out: As covered above, they don’t fire on Web API calls by default. They also don’t cover bulk imports or integration syncs without explicit opt-in at the integration level.
The pattern underneath all of this
Every entry point into Dataverse is a separate duplicate-prevention problem.
Native duplicate detection was designed for the UI, for a user filling in a form and hitting save. That’s the scenario it handles reliably. The Web API, bulk imports, integration syncs, and automated flows are architecturally separate entry points, and each one requires its own solution under the current model.
On a straightforward single-system migration, the header flag or a post-load cleanup might be enough. But CE implementations rarely look like that. Most involve a legacy CRM, at least one integration running at go-live, and a data model with custom entities that nobody built a plugin for. The number of entry points to cover multiplies, and the cost of missing one lands at go-live, or three weeks after, when the client calls.
What a native Dataverse-level layer looks like
Plauti Deduplicate for Dynamics 365 installs directly from Microsoft Marketplace and runs entirely inside the client’s Dataverse environment. No external access, no middleware, no data leaving the environment.
It catches duplicates at every entry point, form saves, bulk imports, Web API calls, integrations syncs, without requiring each one to be configured separately. It works across all standard and custom Dataverse tables, not just Account, Contact, and Lead. There’s no 5,000-record ceiling on bulk jobs.
For partners, the practical difference is that the duplicate workstream gets scoped and handled once, rather than re-solved on every project with a different workaround. The tool is already configured for the next migration before you’ve finished the current one.
What to do next?
If you’re mid-project and need to understand the scope of what’s come through, start with the bulk detection job to get a baseline, keeping the 5,000-record cap in mind. Then work back through the active integrations and check whether they’re passing the detection header.
If you’re at the architecture stage, it’s worth building duplicate prevention into the integration design rather than treating it as a post-migration cleanup task.
And if you want to see how partners handle this across every project with one consistent tool, book a 20-minute meeting walkthrough with the Plauti team. No demo theater, just the specific flows a CE consultant would actually configure.
The post Web API inserts bypass D365 duplicate detection by default. Here’s what to do about it appeared first on CRM Software Blog | Dynamics 365.