Synchronizing content requests

When building an integration a large part is about synchronizing the results of content requests as they become completed.

States

The state of a content request be changed from any value to any other value, which means that a completed request can sometimes become confirmed again.

This means that a content request can be completed more than once and that your integration should plan for this. For example if your integration automatically publishes a page you might want to:

• Keep the page available but update it when the request is completed again.
• Unpublish the page and republish it when the request is completed again.

The exact route to take depends on the goals of your integration.

Getting all request changes

The recommended way to to fetch requests is based on their last state change which allows you to monitor the progress of your requests.

For the HTTP API this means calling the endpoint for querying content with a query that does a few things:

• Sets a criteria to limit results to content requests sent via the API (source = api)
• Sets a criteria that fetches requests where lastStateChange is after the last state change seen by the sync
• Sorts the results by lastStateChange, so that if a sync fails the next sync can restart correctly

The API will paginate the results and an integration can either rely on the page query parameter to fetch several pages or tweak the criteria for completed when a page has been handled.

In pseudo-code a synchronization method might look like this:

while true
  let lastStateChangeSeen = getLastStateChangeSeen()
  let requests = fetchChangedSince(lastStateChangeSeen)

  if requests.length == 0
    // Nothing completed - nothing more to do exit the do-loop
    break
  done

  for request of requests
    storeContentRequest(request)
    lastStateChangeSeen = request.lastStateChange;
  done

  setLastStateChangeSeen(lastStateChangeSeen);
done

Running as synchronization based on such a pseudo-code at a set interval will fetch completed content requests as they become available. The interval can be 5 minutes, 1 hour or even once a day. If you want to handle content requests as they change state the HTTP API supports callbacks via webhooks.

Getting completed requests

If you are only interested in when things are completed the easiest way is to perform a query for requests since either:

1. The last time the integration fetched completed requests
2. The completion time of the last completed content request

Similar to fetching all requests construct a query that:

• Sets a criteria to limit results to content requests sent via the API (source = api)
• Sets a criteria that fetches requests where completed is after the previously picked date
• Sorts the results by completed, so that if a sync fails and option 2 was used the next sync can restart correctly

In pseudo-code this would look something like this:

while true
  let dateOfLastCompleted = getLastCompletedInDatabase()
  let requests = fetchCompletedSince(dateOfLastCompleted)

  if requests.length == 0
    // Nothing changed state - nothing more to do exit the do-loop
    break
  done

  for request of requests
    storeContentRequest(request)
  done
done

Error handling

An import part of building a stable synchronization is error handling, things such as logging when something goes wrong and allowing the synchronization to proceed.

It is recommended that your system at a minimum logs cases where it can't access the API and if it is unable to store a received content request.

In addition to this we recommend pinging us with an import state whenever your system imports a finished content request. Using the HTTP API this can be done via the endpoint for setting import state.

Your integration may ping us both after a successful import and if an error occurs.

In pseudo-code a method for storing content requests might look like:

if request.state == 'completed'
  try
    storeStateAndInfoForRequest(request)

    transferContent(request, targetEntity);

    updateImportState(request, 'success')
  catch ex
    logError(ex)

    updateImportState(request, 'fail')
else
  try
    storeStateAndInfoForRequest(request)
  catch ex
    logError(ex)

In this pseudo-code storeStateAndInfoForRequest is a method that performs an API call to update the import state with a short time out and a catch-all for any errors. This is recommended so that a synchronization does not lock for a long time while waiting for the call to complete and that the call does not break the synchronization if it errors.