Coursedog executes a complex system of integration functions every night to sync the Coursedog system with an SIS. We both GET data out of the SIS and POST updates back through REST API endpoints. Every school for which an SIS integration has been set up will have their integration functions executed on a nightly basis (around 3AM EST time).
Merges are executed in a specific order as defined by the dependencies between different elements (sections before courses, etc..). When there are multiple schools, execution functions are run in batches, so the order in which the function starts is not necessarily the other in which it is completed.
Every school can have custom merge settings, found in the "Merge Settings" tab of the Admin dashboard:
The first setting is "Merge Types" which lists all objects that are being merged. You can turn the integration on/off for any of these data objects.
Users can define the "nightly integration schedule", which allows users to define whether merges are run nightly for enabled merge types.
The "type-specific" settings define the default source of truth, along with field specific exceptions. This setting can also be different for different object types.
The source of truth options are used to define the logic for handling merge conflicts, and are as follows
- Manual resolution - if there is a conflict, stop the integration for that object and return an error
- Resolve as Coursedog - if there is a conflict, take the value from the Coursedog system as the source of truth
- Resolve as Institution - if there is a conflict, take the value from the SIS as the source of truth
- Always Coursedog - take all values from Coursedog (this is the same as not having the integration on at all)
- Always Institution - take all values from SIS
Additionally, for each object, there are field specific exceptions. The main purpose of these exceptions is that the Coursedog system often contains much more functionality than an SIS, and so there can be fields that exist in Coursedog that don't exist in the SIS. To ensure these fields don't get wiped out in the nightly merge, we can specify these exceptions where we set the source of truth for these fields as "Always Coursedog"
Every night when the integration is run, the following steps occur
- We first GET all the school data and run it through our internal data verifier. If there is a verification error, we return an error and stop the merge
- We then GET all of our data
- We GET a copy of prior night's data (stored in our data backups) which represents the last successful merge
- We execute a three way merge - It is a three way merge because we use three data sources to execute the merge. The three sources being
- Their Data
- Our Data
- Yesterday's version of the data
- After executing the merge, if any merge conflicts exist, we store the merge conflicts as errors and stop the merge for all objects that had a conflict. For all other objects, they continue to the next phase
- If a school has POST APIs set up, the POST APIs are executed
- After the POST APIs, we GET their data again because the SIS sometimes adds IDs or other fields to the data, and we want to make sure we have those updated IDs
- If there were no issues with the POST APIs we take the new version of all objects and store it in our database.
- We send out a merge report email
Reasons For Errors
The following are a list of reasons that would lead to errors with the nightly merge
- SIS servers down
- N2N API middleware down
- GET endpoint returning multiple objects with the same ID
- Verification Error
- Merge Conflict
- POST API Error
Identifying Errors - Merge Reports
After each nightly sync, Coursedog automatically produces a merge report, which can be found in the merge dashboard in the admin panel (it is likely that only a few users will have access to this, specifically IT people)
For each merge, you can click on "View Report Details", which takes you to the merge dashboard:
The dashboard shows the status of each step of the integration, along with all errors that occurred that prevented the merge from completing successfully. Most errors will occur during the POST Updates to SIS phase, and report will detail exact error response that your SIS provides to us to help IT folks understand why their SIS is rejecting our update requests.
Potential Error Messages
- "Failing with [#] Errors Sending Back to SIS" - means that a realtime update during the day has failed to successfully sync to the SIS