Using Dynamics365 as a Queue for Data Synchronization

Over the years, I’ve migrated a lot of data from on-premise systems into Dynamics365 (whether they be existing CRM system, homegrown solutions or off the shelf packages).  I’ve used a number of third-party tools to accomplish these tasks (Scribe and Kingsway) but have also written my own when the need arose.

On a recent project, faced with yet more synchronization requests and the need for more infrastructure to manage changes, mediate conflicts, prevent ping-ponging data writes, etc, etc.  I started to change my thinking from being able to have everything on-premise (i.e., the ability to queue up new Virtual Images and tons of server space et al) to think of how I solve this problem if all I had was Dynamics365 and the server I am moving data from.

To start with – how could I keep up-to-date with all the changes happening in Dynamics and queue them up for later retrieval by some other system.

My first thought was an async workflow to do the job but this raised up a few other requirements in doing this;

  1. Administrators should be able to associate this workflow to any new entity they want synced with super ease (i.e., create workflow, finish).
  2. The code for the workflow should not need to be modified at all and should dynamically figure out the entity and primary key attribute that I need to later retrieve to be synced.
  3. Code should be small.

So here is what I wrote as a workflow and then deployed to my online tenant.  The solution is really tiny, I created an entity called syn_dataeventqueue, which contains all the synchronization entries.

I did some tests between custom and core entities and was able to detect the proper change events coming in and for the correct entities.  You can see the initial state is “Not Processed” – I created some custom states for when I pull the requests to not pull again if the syncing period went longer than expected but that’s for another post – here is the code.

 protected override void Execute(CodeActivityContext Execution)
 {
 //Get the Tracing Service
 ITracingService tracingService = Execution.GetExtension<ITracingService>();

 //get context
 IWorkflowContext context = Execution.GetExtension<IWorkflowContext>();
 //create iorganization service object
 IOrganizationServiceFactory serviceFactory = Execution.GetExtension<IOrganizationServiceFactory>();
 IOrganizationService service = serviceFactory.CreateOrganizationService(context.InitiatingUserId);

 //Now we need to query the entity for their primary id.
 RetrieveEntityRequest request = new RetrieveEntityRequest();
 request.EntityFilters = EntityFilters.Attributes;
 request.LogicalName = context.PrimaryEntityName;

 RetrieveEntityResponse response = (RetrieveEntityResponse)service.Execute(request);
 AttributeMetadata PrimaryAttribute = response.EntityMetadata.Attributes.Where(a => a.IsPrimaryId == true && a.AttributeType == AttributeTypeCode.Uniqueidentifier && a.ColumnNumber == 1).FirstOrDefault();
 string AttributeName = PrimaryAttribute.SchemaName.ToLower();

 Entity EntityToSync = (Entity)service.Retrieve(context.PrimaryEntityName, context.PrimaryEntityId, new ColumnSet(AttributeName));

 tracingService.Trace("Trace| Record to Synchronize: Entity [{0}], Id [{1}].", context.PrimaryEntityId.ToString(), context.PrimaryEntityName.ToString());
 

 if (EntityToSync.Contains(AttributeName))
 {
 try
 {
 Entity SyncroEntity = new Entity("syn_dataeventqueue");
 SyncroEntity["syn_name"] = context.PrimaryEntityName;
 SyncroEntity["syn_entityrecordid"] = context.PrimaryEntityId.ToString();
 service.Create(SyncroEntity);
 }
 catch (Exception ex)
 {
 tracingService.Trace("Error| Synchronization Submission Exception: {0}", ex.ToString());
 }
 }

The only piece you are probably wonder about is the search for ColumnNumber to be 1.  In my tests, this is always the primary id field, when I tried searching for simply the IsPrimaryId it brought back results for the primary ids of related entities so that didn’t work.

Here is how things look in the workflow creation itself.

wf.PNG

One step!

And what does it look like in Dynamics?

grid

Beautiful – now to finish the rest of it.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s