Coursedog

Submit a Ticket My Tickets
Welcome
Login  Sign up

Nightly Integration Overview

Table of Contents

Overview
Merge Duration
Merge Order
Merge Settings
Integration Pipeline
Reasons For Errors
Identifying Errors - Merge Reports
Related Articles


Overview

  • Coursedog executes a complex system of integration functions every night to sync the Coursedog system with your SIS. 

  • We both GET data out of the SIS and POST updates back through REST API endpoints.

  • Every school for which an SIS integration has been set up will have their integration functions executed on a nightly basis. 


Merge Duration

  • Start and end time can vary; if you have any questions about the merge duration for your institution, please reach out to Coursedog. 

  • When a nightly merge ends depends on a variety of factors, including the time it starts, whether or not the institution or SIS is undergoing maintenance/backup, whether or not your institution consists of multiple schools, and whether your nightly integration queue is not progressing due to issues in other instances.

  • When the merge is running, users will see a message indicating that a sync is in progress.

  • It is not common, but occasionally nightly merges do run into business hours. 


Merge Order

Merges are executed in a specific order as defined by the dependencies between different elements (sections before courses, etc.). When a single institution has multiple schools, execution functions are run in batches, so the order in which the function starts is not necessarily the order in which it is completed. 


Merge Settings

Overview | Merge Types | Nightly Integration Schedule | Type-Specific Settings | Execute Merge

PATH: Admin > Merge Settings

  • Actual settings can vary by SIS and institution.

  • What you see below is a general overview; please work with your Customer Success representative to confirm the correct settings for you. They will work with a data engineer (DE) to modify as needed. 


Merge Types

The first setting is "Merge Types" which lists all entities that are being merged. You can enable (set to “Yes”) or disable (set to “No”) the integration for each of these data objects.


Nightly Integration Schedule

  • Users can define the "nightly integration schedule". 

  • When set to “Yes”, merges will run nightly for all enabled entities.

  • When set to “No”, nightly merges will NOT be run.

Type-Specific Settings

Overview

  • The "type-specific" settings define the default source of truth for the selected entity, along with field-specific exceptions. 

  • The “Source of truth” defines the logic for handling merge conflicts. 

  • Click through all of the different object types/entities at top to configure source-of-truth settings for each.


Source of Truth Options

Manual Resolution | Resolve as Coursedog | Resolve as Institution
Always Coursedog | Always Institution

Manual Resolution

If there is a conflict, stop the integration for that object and return an error.


Resolve as Coursedog

  • If there is a conflict, take the value from the Coursedog system as the source of truth.

  • Use this setting if a data type exists in both Coursedog and the SIS but it is maintained in Coursedog (and therefore you know that Coursedog is always the most current). 

  • This is useful for bidirectional integrations where we can pull information from your SIS; store it in Coursedog; edit it in Coursedog; and then post back to the SIS. 

  • There are many potential use cases for this – essentially, any field that exists in both your SIS and Coursedog but that is maintained in Coursedog. 


Resolve as Institution

If there is a conflict, take the value from the SIS as the source of truth.


Always Coursedog


Always Institution

Take all values from SIS.



Field-Specific Exceptions

  • Additionally, for each object, there can be field-specific exceptions.

  • The main purpose of these exceptions is that the Coursedog system often contains much more functionality than an SIS, and so there can be fields that exist in Coursedog that don't exist in the SIS.

  • To ensure these fields don't get wiped out in the nightly merge, we can add these exceptions to the “field” input and then set the source of truth for these exceptions as "Always Coursedog".

  • In other words, if the default source of truth for “Rooms” is “Always Institution” – but there are some room fields that have a different source of truth – you can add those fields as “Field-specific exceptions”. 

Execute Merge

  • In nightly merges updates won’t be sent to the SIS when there are more than 100 entities with changes (this is a protective measure to prevent unintended updates).

  • If you need to update more than 100 entities, you can run a manual merge at Admin > Execute Merge with this protection turned off. Learn more here

Integration Pipeline

Every night when the integration is run, the following occurs: 


  1. We first GET all the school data and run it through our internal data verifier. If there is a verification error, we return an error and stop the merge.

  2. We then GET all of our data.

  3. We GET a copy of the previous night's data (stored in our data backups) which represents the last successful merge.

  4. We execute a three-way merge because we use three data sources to execute the merge. The three sources being: the institution’s data, Coursedog’s data, and yesterday’s version of the data. 

  5. After executing the merge, if any merge conflicts exist, we store the merge conflicts as errors and stop the merge for all objects that had a conflict. For all other objects, they continue to the next phase.

  6. If the institution has POST APIs set up, the POST APIs are executed.

  7. After the POST APIs, we GET the institution’s data again because the SIS sometimes adds IDs or other fields to the data, and we want to make sure we have those updated IDs.

  8. If there were no issues with the POST APIs, we take the new version of all objects and store it in our database.

  9. We send out a merge report email.


Reasons For Errors

The following issues can lead to errors with a nightly merge:

  • SIS servers down

  • API middleware down

  • GET endpoint returning multiple objects with the same ID

  • Verification Error

  • Merge Conflict

  • POST API Error


Identifying Errors - Merge Reports

You can review merge errors in multiple places: via email notifications; the admin panel; on the Scheduling reports dashboard; and in the Section Dashboard. You can learn more about managing integration errors here


Related Articles

Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.