Table of Contents
Overview of Integration Merge Pipeline
How Merges Work
Merge Setting Configurability
Examples of Merges
Field-Specific Exceptions
Term-Specific Integration Objects
Integration Pipeline Steps
Integration Dry Run
Integration Schedule - Real Time
Merge Reports
Related Articles
Overview of Integration Merge Pipeline
All merge types result from a specific REST API endpoint that Coursedog can access. Each integration merge follows a series of very specific steps. These steps are:
We GET all the school data using a REST API
We GET all Coursedog data
We verify all school data is formatted correctly
We verify all Coursedog data is formatted correctly
We merge Coursedog data with SIS data (see notes below on how merging works)
If any merge conflicts exist, we store the merge conflicts as errors and stop the merge for all objects that had a conflict.
For all other objects, they continue to the next phase.
If a school has POST APIs set up, updates are sent from Coursedog to SIS
We store results of the merge in Coursedog
We save a merge report to be viewed in Coursedog
How Merges Work
When Coursedog merges our data with SIS data, we first get the following data sources:
Coursedog Data
SIS Data
Output of the last successful merge (which we will call Original Data)
We use these three data sources to perform a three-way merge, a process which is best explained with an example. Let's say we have the following data:
{
original: 1,
sis: 1,
coursedog: 2
}
The three-way merge will compare the Coursedog value to the SIS value and to the original. It detects that:
Coursedog is different from SIS
Coursedog is different from original
SIS and original equal each other
As a result, it will accept that a change was made on the Coursedog side, and store the change as an update.
However, let's say we have the following data:
{
original: 1,
sis: 3,
coursedog: 2
}
The three-way merge detects that:
Coursedog is different from the SIS.
Coursedog is different from the original.
SIS is different from the original.
The only difference in this example and the previous one is that the SIS value has also changed. So even though the Coursedog value changed as well, our system now does not know which change is supposed to be accepted. Is the value supposed to be 3 or is it supposed to be 2? The System does not know, so it will return this as a merge conflict because both the SIS data and Coursedog data changed, and neither match with the original.
Merge Setting Configurability
For each merge type, custom settings can be configured to define the behavior of the merge and the logic for specific fields.
The default source of truth is used to determine the behavior of a merge, and how to handle merge conflicts.
Manual Resolution – If there is a conflict, stop the integration for that object and return an error.
Resolve as Coursedog – If there is a conflict, take the value from the Coursedog system as the source of truth. This option should only be used during bi-directional integration.
Resolve as Institution – If there is a conflict, take the value from the SIS as the source of truth. This option should only be used during bi-directional integration.
Always Coursedog –Take all values from Coursedog.
Always Institution – Take all values from SIS.
Examples of Merges
Merges Set to Resolve on a Nightly Bi-Directional Merge
Day 1: Data is retrieved from the SIS and brought into Coursedog.
Day 2: Coursedog should be the source of truth for Sections. Updates are made in Coursedog.
Section data is retrieved from the SIS
A difference is detected (because we made changes in Coursedog)
Because the source of truth is Coursedog, we keep the Coursedog change and drop the SIS value
Because we are bi-directional, we send the Coursedog update to the SIS
We perform a backup of the current state of Coursedog and SIS data
Day 3: Coursedog should be the source of truth for Sections. Section updates are made in SIS.
Section data is retrieved from the SIS
A difference is detected (because we made changes in the SIS)
Because the source of truth is Coursedog, we keep the Coursedog change and drop the SIS change
Because we are bi-directional, we send the Coursedog update to the SIS (again)
We perform a backup of the current state of Coursedog and SIS data
Merges Set to Resolve on a One-Way Sync
Day 1: Data is retrieved from the SIS and brought into Coursedog.
Day 2: Coursedog should be the source of truth for Sections. Section updates are made in Coursedog.
Section data is retrieved from the SIS
A difference is detected (because we made changes in Coursedog)
Because the source of truth is Coursedog, we keep the Coursedog change and drop the SIS value
Because we are one-way, we do not send the Coursedog update to the SIS
We perform a backup of the current state of Coursedog and SIS data
Day 3: Coursedog should be the source of truth for Sections. Section updates are made in SIS.
Section data is retrieved from the SIS
A difference is detected (because we did not push our changes to the SIS, so the SIS still has the old value)
No conflict is detected (because we did not change our record today)
Because no conflict was detected, but the SIS data is different than our data (and therefore newer) we drop the Coursedog change and merge in the SIS change
Because we are one-way, we don’t send the Coursedog update to the SIS
We perform a backup of the current state of Coursedog and SIS data
Day 4: Coursedog appears to be missing changes made locally for the past 2 days.
Field-Specific Exceptions
Overview
Field-specific exceptions are used to define custom logic for custom fields.
This is most often used to identify data fields that don't exist in the SIS, and so as a result must be marked as "Always Coursedog"; otherwise, these data fields will be erased during any merge.
Note on Revisions in Curriculum
Overview
When a new course or program revision is created in the SIS and sent to Coursedog, by default all Coursedog-owned fields and data in those fields will copy over into the new revision.
By persisting these Coursedog-owned fields from one revision to the next, we ensure that critical fields, such as outcomes or requirements, are not wiped when a new revision is created in the SIS.
In turn, this allows for business processes such as bulk revisioning to occur in the SIS without negative impact on data housed in Coursedog.
You can select which Coursedog-owned fields to persist under Admin Panel > New Revision Behavior.
Example
Coursedog: [Course Title], [Learning Outcomes]
Revision from the SIS: [Course Title]
New Coursedog Revision: [NEW Course Title], [Learning Outcomes]
Effective Dating
Overview
This behavior interacts with effective dating as follows: The data which persists is from the most-recent, but not future-dated, data compared to the SIS revision.
Scenario 1
This means that in the below scenario, requirements from the Coursedog 2000 revision would copy forward into the revision which originated from the SIS in 2005.
Scenario 2
In the below scenario, the January 2003 revision is the most recent revision in Coursedog, and it does not include requirements. Therefore, the Jan 2005 revision which originated in the SIS will not include requirements (persisting/copying forward doesn’t work for the empty field).
Scenario 3
In the below scenario, the revision in Coursedog which houses the requirements is after the SIS originated revision. Therefore, the resultant net-new Coursedog revision does not include requirements.
Term-Specific Integration Objects
For integration objects that are term specific (Sections, Courses, Relationships), you can specify the allowed terms for the nightly integration. If this field is populated, the nightly integrations will only execute for the specified terms. If no terms are specified, integrations will be run for the current active term and all future terms.
Integration Pipeline Steps
Each of the integration pipeline steps can be turned on/off at any point. However, the only step that we would suggest turning off at some points would be the "Should Coursedog send updates to the SIS?" step (see “Integration Dry Run” below to learn more about why you might want to sometimes set this step to “NO”).
Integration Dry Run
If you’d like to run a merge that captures all the POSTs that would be triggered without actually sending anything to your SIS, you will need to make the following changes to your settings under “Integration steps to execute”:
Set “Should Coursedog send updates to the SIS?” to “NO”. When this is set to “NO”, a sub-toggle will appear that will ask “Should updates be stored in merge report?”
Set “Should updates be stored in merge report” to “YES”. Updates will then be generated and included in the merge report but not sent to the SIS.
Set “Should Coursedog save merged data” to “NO”.
Use Cases
An integration dry run is helpful in the following use cases:
You are adding a term to integration scope and want to figure out if anything would be sent back.
You previously locked/removed a term from integration, are re-enabling it, and want to determine the scope of the first merge from Coursedog to the SIS to see how many changes were made in Coursedog before proceeding.
You made a change to a template, merge settings, or other configuration and want to determine the impact to the integration without sending changes to the SIS.
Integration Schedule - Real Time
Overview
We have the ability to turn on a real-time integration for some of our integrations. This means when a user saves changes to an existing section, the real-time sync process will be initiated.
Only Coursedog administrators can edit these settings.
When real-time syncs are turned on, you can specify how often the data should be synced in aggregate. This is present because a true "real-time" sync might overload the school's system, so Coursedog will actually queue up updates, and then execute them in bulk after a preset delay, which is defined by the "How Often Should Changed Data Be Synced" setting - which defaults to 30 minutes.
You can learn more about suggested settings for real-time merges – and how they work – here.
Merge Reports
Overview
After each merge, a merge report will be stored, which will show all steps of the merge that executed, along with any errors that resulted. Merge reports are found in the Admin Panel.
Each merge report will provide information on each step that was executed.
Merge Errors Overview
Within each step, you can view details about the step along with any errors:
If there are any merge conflicts, those errors will exist in the "Merge SIS And Coursedog Data" step. A sample merge error is as follows:
{
"COM1075131409": [
{
"kind": "E",
"lhs": 4,
"rhs": 0,
"path": [
"maxEnrollment"
]
}
]
}
Merge Error Causes
There are four fields for a merge conflict to explain what caused the issue:
Kind – The options are N,D,E
N - New (a field was added to the SIS data)
D - Deleted (a field was deleted from the SIS data)
E - Edited (a field was edited in both SIS and Coursedog data)
LHS – This stands for Left Hand Side, which is the Coursedog data.
RHS – This stands for Right Hand Side, which is the SIS data.
Path – This refers to the path to access the data in the data object.