Table of Contents
Overview
Merge Duration
Merge Order
Merge Settings
Integration Pipeline
Reasons For Errors
Identifying Errors - Merge Reports
Related Articles
Overview
Coursedog executes a complex system of integration functions every night to sync the Coursedog system with your SIS.
We both GET data out of the SIS and POST updates back through REST API endpoints.
Every school for which an SIS integration has been set up will have their integration functions executed on a nightly basis.
Merge Duration
Start and end time can vary; if you have any questions about the merge duration for your institution, please reach out to Coursedog.
When a nightly merge ends depends on a variety of factors, including the time it starts, whether or not the institution or SIS is undergoing maintenance/backup, whether or not your institution consists of multiple schools, and whether your nightly integration queue is not progressing due to issues in other instances.
When the merge is running, users will see a message indicating that a sync is in progress.
It is not common, but occasionally nightly merges do run into business hours.
Merge Order
Merges are executed in a specific order as defined by the dependencies between different elements (sections before courses, etc.). When a single institution has multiple schools, execution functions are run in batches, so the order in which the function starts is not necessarily the order in which it is completed.
Merge Settings
Overview | Merge Types | Nightly Integration Schedule | Type-Specific Settings | Execute Merge
PATH: Admin > Merge Settings
Actual settings can vary by SIS and institution.
What you see below is a general overview; please work with your Customer Success representative to confirm the correct settings for you. They will work with a data engineer (DE) to modify as needed.
Merge Types
The first setting is "Merge Types" which lists all entities that are being merged. You can enable (set to “Yes”) or disable (set to “No”) the integration for each of these data objects.
Nightly Integration Schedule
Users can define the "nightly integration schedule".
When set to “Yes”, merges will run nightly for all enabled entities.
When set to “No”, nightly merges will NOT be run.
Type-Specific Settings
Overview
The "type-specific" settings define the default source of truth for the selected entity, along with field-specific exceptions.
The “Source of truth” defines the logic for handling merge conflicts.
Click through all of the different object types/entities at top to configure source-of-truth settings for each.
Source of Truth Options
Manual Resolution | Resolve as Coursedog | Resolve as Institution
Always Coursedog | Always Institution
Manual Resolution
If there is a conflict, stop the integration for that object and return an error.
Resolve as Coursedog
If there is a conflict, take the value from the Coursedog system as the source of truth.
Use this setting if a data type exists in both Coursedog and the SIS but it is maintained in Coursedog (and therefore you know that Coursedog is always the most current).
This is useful for bidirectional integrations where we can pull information from your SIS; store it in Coursedog; edit it in Coursedog; and then post back to the SIS.
There are many potential use cases for this – essentially, any field that exists in both your SIS and Coursedog but that is maintained in Coursedog.
Resolve as Institution
If there is a conflict, take the value from the SIS as the source of truth.
Always Coursedog
Take all values from Coursedog (this is the same as not having the integration on at all).
Use this setting if this data type doesn’t exist in the SIS. This will help prevent data from being wiped in a merge.
A potential use case for this setting would be department dynamic steps, requirement levels, Course Ids, Section Numbers, and Last Edited At Relationship Fields. Other example use cases are outlined here, here, here.
Always Institution
Take all values from SIS.
Field-Specific Exceptions
Additionally, for each object, there can be field-specific exceptions.
The main purpose of these exceptions is that the Coursedog system often contains much more functionality than an SIS, and so there can be fields that exist in Coursedog that don't exist in the SIS.
To ensure these fields don't get wiped out in the nightly merge, we can add these exceptions to the “field” input and then set the source of truth for these exceptions as "Always Coursedog".
In other words, if the default source of truth for “Rooms” is “Always Institution” – but there are some room fields that have a different source of truth – you can add those fields as “Field-specific exceptions”.
Execute Merge
In nightly merges updates won’t be sent to the SIS when there are more than 100 entities with changes (this is a protective measure to prevent unintended updates).
If you need to update more than 100 entities, you can run a manual merge at Admin > Execute Merge with this protection turned off. Learn more here.
Integration Pipeline
Every night when the integration is run, the following occurs:
We first GET all the school data and run it through our internal data verifier. If there is a verification error, we return an error and stop the merge.
We then GET all of our data.
We GET a copy of the previous night's data (stored in our data backups) which represents the last successful merge.
We execute a three-way merge because we use three data sources to execute the merge. The three sources being: the institution’s data, Coursedog’s data, and yesterday’s version of the data.
After executing the merge, if any merge conflicts exist, we store the merge conflicts as errors and stop the merge for all objects that had a conflict. For all other objects, they continue to the next phase.
If the institution has POST APIs set up, the POST APIs are executed.
After the POST APIs, we GET the institution’s data again because the SIS sometimes adds IDs or other fields to the data, and we want to make sure we have those updated IDs.
If there were no issues with the POST APIs, we take the new version of all objects and store it in our database.
We send out a merge report email.
Reasons For Errors
The following issues can lead to errors with a nightly merge:
SIS servers down
API middleware down
GET endpoint returning multiple objects with the same ID
Verification Error
Merge Conflict
POST API Error
Identifying Errors - Merge Reports
You can review merge errors in multiple places: via email notifications; the admin panel; on the Scheduling reports dashboard; and in the Section Dashboard. You can learn more about managing integration errors here.