Coursedog

Submit a ticket My Tickets
Welcome
Login  Sign up

How Merges Work in Coursedog

Overview of Integration Merge Pipeline


All merge types result from a specific REST API endpoint that can Coursedog can access. 


Each integration merge follows a series of very specific steps. These steps are

  1. We GET all the school data using a REST API
  2. We GET all Coursedog data
  3. We verify all school data is formatted correctly
  4. We verify all Coursedog data is verified correctly
  5. We merge Coursedog data with SIS data (see notes below on how merging works)
    1. If any merge conflicts exist, we store the merge conflicts as errors and stop the merge for all objects that had a conflict. For all other objects, they continue to the next phase
  6. If a school has POST APIs set up, updates are sent from Coursedog to SIS
  7. We store result of merge in Coursedog
  8. We save a merge report to be viewed in Coursedog



How Merges Work



For more information on the merge technical architecture, please visit Merge Technical Architecture.


When Coursedog merges our data with SIS data, the following occurs:

We first get the following data sources:

  1. Coursedog Data
  2. SIS Data
  3. Output of the last successful merge (which we will call Original Data)

We use these three data sources to perform a three-way merge, a process which is best explained with an example. Let's say we have the following data:

{
original: 1,
sis: 1,
coursedog: 2
}

The three way merge will compare the Coursedog value to the SIS value and to the original. It detects that:

  1. Coursedog is different from SIS
  2. Coursedog is different from original
  3. SIS and original equal each other

As a result, it will accept that a change was made on the Coursedog side, and store the change as an update.

However, let's say we have the following data:

{
original: 1,
sis: 3,
coursedog: 2
}

The three way merge detects that the following

  1. Coursedog is different from the SIS
  2. Coursedog is different from the original
  3. SIS is different from the original

The only difference in this example and the one above is that the SIS value has also changed. So even though the Coursedog value changed as well, our system now does not know which change is supposed to be accepted. Is the value supposed to be 3 or is it supposed to be 2? The System does not know, so it will return this as a merge conflict because both the SIS data and Coursedog data changed, and neither match with the original.


Merge Setting Configurability


For each merge type, custom settings can be configured to define the behavior of the merge, and the logic for specific fields.


The default source of truth is used to determine the behavior of a merge, and how to handle merge conflicts

  1. Manual resolution - if there is a conflict, stop the integration for that object and return an error
  2. Resolve as Coursedog - if there is a conflict, take the value from the Coursedog system as the source of truth - NOTE: This option should only be used during bi-directional integration.
  3. Resolve as Institution - if there is a conflict, take the value from the SIS as the source of truth - NOTE: This option should only be used during bi-directional integration.
  4. Always Coursedog - take all values from Coursedog
  5. Always Institution - take all values from SIS


Examples of Merges


Here is an example for when merges are set to resolve as on a nightly bi-directional merge.  


Day 1: Data is retrieved from the SIS and brought into Cd


Day 2: Cd should be the source of truth for Sections. Updates are made in Cd.

  • Section data is retrieved from the SIS
  • A difference is detected (because we made changes in Cd)
  • Because of source of truth is Cd, we keep the Cd change and drop the SIS change
  • Because we are bi-directional, we send the Cd update to the SIS
  • We perform a backup of the current state of Cd and SIS data

Day 3: Cd should be the source of truth for Sections. Section updates are made in SIS.

  • Section data is retrieved from the SIS
  • A difference is detected (because we made changes in the SIS)
  • Because of source of truth is Cd, we keep the Cd change and drop the SIS change
  • Because we are bi-directional, we send the Cd update to the SIS (again)
  • We perform a backup of the current state of Cd and SIS data


Here is another example for when merges are set to resolve as... on a one way sync:


Day 1: Data is retrieved from the SIS and brought into Cd


Day 2: Cd should be the source of truth for Sections. Section updates are made in Cd.

  • Section data is retrieved from the SIS
  • A difference is detected (because we made changes in Cd)
  • Because of source of truth is Cd, we keep the Cd change and drop the SIS change
  • Because we are one-way, we do not send the Cd update to the SIS
  • We perform a backup of the current state of Cd and SIS data

Day 3: Cd should be the source of truth for Sections. Section updates are made in SIS.

  • Section data is retrieved from the SIS
  • A difference is detected (because we did not push our changes to the SIS, so the SIS still has the old value)
  • No conflict is detected (because we did not change our record today)
  • Because no conflict was detected, but the SIS data is different than our data (and therefore newer) we drop the Cd change and merge in the SIS change
  • Because we are one-way, we dont send the Cd update to the SIS
  • We perform a backup of the current state of Cd and SIS data


Day 4: Cd appears to be missing changes made locally for the past 2 days.



Field-Specific exceptions are used to define custom logic for custom fields. This is most often used to identify data fields that don't exist in the SIS, and so as a result must be marked as "Always Coursedog", otherwise these data fields will be erased during any merge.


For integration objects that are term specific (Sections, Courses, Relationships), you can specify the allowed terms for the nightly integration:


If this field is populated, the nightly integrations will only execute for the specified terms. If not terms are specified, integrations will be run for the current active term and all future terms.


Each of the integration pipeline steps can be turned on/off at any point.


The only step that we would suggest turning off at some points would be the "Should Coursedog send updates to the SIS" step - which when turned off cause the integration to NOT send any updates to the SIS.



Integration Schedule - Real Time




We have the ability to turn on a real time integration for some of our integrations. This means when a user saves changes to an existing section, the real time sync process will be initiated. Only Coursedog administrators can edit these settings, and we would not recommend turning on without fully understanding the behavior.

Real time syncs are best used when Coursedog stands to be the source of truth, and Coursedog does not expect to ever receive data updates from the SIS. Additionally, Real time syncs are more efficient when Coursedog does not need to GET data from the SIS to be able to execute a successfully update.

When real-time syncs are turned on, you can specify how often the data should be synced. This is present because a true "real-time" sync might overload the school's system, so Coursedog will actually queue up updates, and then execute them in bulk after a preset delay, which is defined by the "How Often Should Changed Data By Synced" setting - which defaults to 30 minutes.

Additionally, for real-time syncs, you can override default settings for which steps to execute. If Coursedog is the source of truth for data that will be synced to SIS - we suggest turning off all settings except

  1. Should Coursedog use its own data
  2. Should Coursedog verify its own data
  3. Should Coursedog send updates to the SIS?

These settings tell the merge to ignore what is in the SIS, take only updates from our data, and send those updates to the SIS.


Merge Reports

After each merge, a merge report will be stored, which will show all steps of the merge that executed, along with any errors that resulted. Merge reports are found in the Admin Panel

Each merge report will provide information on each step that executed.

Within each step, you can view details about the step along with any errors:


If there are any merge conflicts, those errors will exist in the "Merge SIS And Coursedog Data" step. A sample merge error is as follows:

{
"COM1075131409": [
    {
      "kind": "E",
      "lhs": 4,
      "rhs": 0,
      "path": [
        "maxEnrollment"
      ]
    }
  ]
}

There are four fields for a merge conflict to explain what caused the issue

  1. Kind. The options are N,D,E
    1. N - New (a field was added to the SIS data)
    2. D - Deleted (a field was deleted from the SIS data
    3. E - Edited (a field was edited in both SIS and Coursedog data)
  2. LHS - This stands for Left Hand Side, which is the Coursedog data
  3. RHS - This stands for Right Hand Side, which is the SIS data
  4. Path - this refers to the path to access the data in the data object


For more information on the merges technical architecture, please visit Merge Technical Architecture.


Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.