Table of Contents
Overview of Integration Merge Pipeline
How Merges Work
Merge Setting Configurability
Examples of Merges
Field-Specific Exceptions
Term-Specific Integration Objects
Integration Schedule - Real Time
Merge Reports
Related Articles
Overview of Integration Merge Pipeline
All merge types result from a specific REST API endpoint that Coursedog can access. Each integration merge follows a series of very specific steps. These steps are:
We GET all the school data using a REST API
We GET all Coursedog data
We verify all school data is formatted correctly
We verify all Coursedog data is formatted correctly
We merge Coursedog data with SIS data (see notes below on how merging works)
If any merge conflicts exist, we store the merge conflicts as errors and stop the merge for all objects that had a conflict.
For all other objects, they continue to the next phase.
If a school has POST APIs set up, updates are sent from Coursedog to SIS
We store results of the merge in Coursedog
We save a merge report to be viewed in Coursedog
How Merges Work
When Coursedog merges our data with SIS data, we first get the following data sources:
Coursedog Data
SIS Data
Output of the last successful merge (which we will call Original Data)
We use these three data sources to perform a three-way merge, a process which is best explained with an example. Let's say we have the following data:
{
original: 1,
sis: 1,
coursedog: 2
}
The three-way merge will compare the Coursedog value to the SIS value and to the original. It detects that:
Coursedog is different from SIS
Coursedog is different from original
SIS and original equal each other
As a result, it will accept that a change was made on the Coursedog side, and store the change as an update.
However, let's say we have the following data:
{
original: 1,
sis: 3,
coursedog: 2
}
The three-way merge detects that:
Coursedog is different from the SIS.
Coursedog is different from the original.
SIS is different from the original.
The only difference in this example and the previous one is that the SIS value has also changed. So even though the Coursedog value changed as well, our system now does not know which change is supposed to be accepted. Is the value supposed to be 3 or is it supposed to be 2? The System does not know, so it will return this as a merge conflict because both the SIS data and Coursedog data changed, and neither match with the original.
Merge Setting Configurability
For each merge type, custom settings can be configured to define the behavior of the merge and the logic for specific fields. The default source of truth is used to determine the behavior of a merge, and how to handle merge conflicts.
Manual Resolution – If there is a conflict, stop the integration for that object and return an error.
Resolve as Coursedog – If there is a conflict, take the value from the Coursedog system as the source of truth. This option should only be used during bi-directional integration.
Resolve as Institution – If there is a conflict, take the value from the SIS as the source of truth. This option should only be used during bi-directional integration.
Always Coursedog –Take all values from Coursedog.
Always Institution – Take all values from SIS.
Examples of Merges
Merges Set to Resolve on a Nightly Bi-Directional Merge
Day 1: Data is retrieved from the SIS and brought into Coursedog.
Day 2: Coursedog should be the source of truth for Sections. Updates are made in Coursedog.
Section data is retrieved from the SIS
A difference is detected (because we made changes in Coursedog)
Because the source of truth is Coursedog, we keep the Coursedog change and drop the SIS value
Because we are bi-directional, we send the Coursedog update to the SIS
We perform a backup of the current state of Coursedog and SIS data
Day 3: Coursedog should be the source of truth for Sections. Section updates are made in SIS.
Section data is retrieved from the SIS
A difference is detected (because we made changes in the SIS)
Because the source of truth is Coursedog, we keep the Coursedog change and drop the SIS change
Because we are bi-directional, we send the Coursedog update to the SIS (again)
We perform a backup of the current state of Coursedog and SIS data
Merges Set to Resolve on a One-Way Sync
Day 1: Data is retrieved from the SIS and brought into Coursedog.
Day 2: Coursedog should be the source of truth for Sections. Section updates are made in Coursedog.
Section data is retrieved from the SIS
A difference is detected (because we made changes in Coursedog)
Because the source of truth is Coursedog, we keep the Coursedog change and drop the SIS value
Because we are one-way, we do not send the Coursedog update to the SIS
We perform a backup of the current state of Coursedog and SIS data
Day 3: Coursedog should be the source of truth for Sections. Section updates are made in SIS.
Section data is retrieved from the SIS
A difference is detected (because we did not push our changes to the SIS, so the SIS still has the old value)
No conflict is detected (because we did not change our record today)
Because no conflict was detected, but the SIS data is different than our data (and therefore newer) we drop the Coursedog change and merge in the SIS change
Because we are one-way, we don’t send the Coursedog update to the SIS
We perform a backup of the current state of Coursedog and SIS data
Day 4: Coursedog appears to be missing changes made locally for the past 2 days.
Field-Specific Exceptions
Overview
Field-specific exceptions are used to define custom logic for custom fields.
This is most often used to identify data fields that don't exist in the SIS, and so as a result must be marked as "Always Coursedog"; otherwise, these data fields will be erased during any merge.
Note on Revisions in Curriculum
Overview
When a new course or program revision is created in the SIS and sent to Coursedog, by default all Coursedog-owned fields and data in those fields will copy over into the new revision.
By persisting these Coursedog-owned fields from one revision to the next, we ensure that critical fields, such as outcomes or requirements, are not wiped when a new revision is created in the SIS.
In turn, this allows for business processes such as bulk revisioning to occur in the SIS without negative impact on data housed in Coursedog.
You can select which Coursedog-owned fields to persist under Admin Panel > New Revision Behavior.
Example
Coursedog: [Course Title], [Learning Outcomes]
Revision from the SIS: [Course Title]
New Coursedog Revision: [NEW Course Title], [Learning Outcomes]
Effective Dating
Overview
This behavior interacts with effective dating as follows: The data which persists is from the most-recent, but not future-dated, data compared to the SIS revision.
Scenario 1
This means that in the below scenario, requirements from the Coursedog 2000 revision would copy forward into the revision which originated from the SIS in 2005.
Scenario 2
In the below scenario, the January 2003 revision is the most recent revision in Coursedog, and it does not include requirements. Therefore, the Jan 2005 revision which originated in the SIS will not include requirements (persisting/copying forward doesn’t work for the empty field).
Scenario 3
In the below scenario, the revision in Coursedog which houses the requirements is after the SIS originated revision. Therefore, the resultant net-new Coursedog revision does not include requirements.
Term-Specific Integration Objects
For integration objects that are term specific (Sections, Courses, Relationships), you can specify the allowed terms for the nightly integration. If this field is populated, the nightly integrations will only execute for the specified terms. If no terms are specified, integrations will be run for the current active term and all future terms.
Integration Schedule - Real Time
Overview
We have the ability to turn on a real-time integration for some of our integrations. This means when a user saves changes to an existing section, the real-time sync process will be initiated.
Only Coursedog administrators can edit these settings.
When real-time syncs are turned on, you can specify how often the data should be synced in aggregate. This is present because a true "real-time" sync might overload the school's system, so Coursedog will actually queue up updates, and then execute them in bulk after a preset delay, which is defined by the "How Often Should Changed Data Be Synced" setting.
You can learn more about suggested settings for real-time merges – and how they work – here.
Merge Reports
Overview
After each merge, a merge report will be stored, which will show all steps of the merge that executed, along with any errors that resulted. Merge reports are found in the Integrations Hub.
Each merge report will provide information on each step that was executed, allowing you to:
Glance at the number of Coursedog Entities that were Updated
Glance at the number of SIS changes that were made
View the duration and time of the merge.
You can also view “Notices”, which is a list of each of the errors that came up during the merge.
Under “Stages”, you can view each step of the merge process that ran, and if necessary, download the logs from this step.
You can also review the changes that took place under the “Updates” tab.
Review the configuration that was used when the merge was executed under the “Configuration” tab.
And review backup files of data before, during, and after merges when needed.
