Skip to main content
Version: Latest

Publications

To ensure that the UI of Sofie is reactive, we are leveraging publications over the DDP connection that Meteor provides.
In its most basic form, this allows for streaming MongoDB document updates as they happen to the UI, and there is also a structure in place for 'Custom Publications' which appear like a MongoDB collection to the client, but are generated in-memory collections of data allowing us to do some processing of data before publishing it to the client.

It is possible to subscribe to these publications outside of Meteor, but we have not found any maintained ddp clients, except for the one we are using in server-core-integration. The protocol is simple and stable and has documentation on the Meteor GitHub, and should be easy to implement in another language if desired.

All of the publication implementations reside in meteor/server/publications folder, and are typically pretty well isolated from the rest of the code we have in Meteor.

We prefer using publications in Sofie over polling because:

  • there are not enough DDP clients to a single Sofie installation for the number of connected clients to be problematic
  • polling can be costly for many of these publications without some form of caching or tracking changes (which starts to get to a similar level of complexity)
  • we can be more confident that all the clients have the same data as the database is our point of truth
  • the system can be more reactive as changes are pushed to interested parties with minimal intervention

MongoDB Publications​

A majority of data is sent to the client utilising Meteor's ability to publish a MongoDB cursor. This allows us to run a MongoDB query on the backend, and let it handle the publishing of individual changes.

In some (typically older) publications, we let the client specify the MongoDB query to use for the subscription, where we perform some basic validation and authentication before executing the query.

In typically newer publications, we are formalising the publications a bit better by requiring some simpler parameters to the publication, with the query then generated on the backend. This will help us ensure that the queries are made with suitable indices, and to ensure that subscriptions are deduplicated where possible.

Custom Publications​

There has been a recent push towards using more 'custom' publications for streaming data to the UI. While we are unsure if this will be beneficial for every publication, it is really beneficial for others as it allows us to do some pre-computation of data before sending it to the client.

To achieve this, we have an optimisedObserver flow which is designed to help maange to a custom publication, with a few methods to fill in to setup the reactivity and the data transformation.

One such publication is the PieceContentStatus, prior to version 1.50, this was computed inside the UI.
A brief overview of this publication, is that it looks at each Piece in a Rundown, and reports whether the Piece is 'OK'. This check is primarily focussed on Pieces containing clips, where it will check the metadata generated by either package manager or media manager to ensure that the clip is marked as being ready for playout, and that it has the correct format and some other quality checks.

To do this on the client meant needing to subscribe to the whole contents of a couple of MongoDB collections, as it is not easy to determine which documents will be needed until the check is being run. This caused some issues as these collections could get rather large. We also did not always have every Piece loaded in the UI, so had to defer some of the computation to the backend via polling.

This makes it more suitable for a custom publication, where we can more easily and cheaply do this computation without being concerned about causing UI lockups and with less concern about memory pressure. Performing very granular MongoDB queries is also cheaper. The result is that we build a graph of what other documents are used for the status of each Piece, so we can cheaply react to changes to any of those documents, while also watching for changes to the pieces.

Live Status Gateway​

The Live Status Gateway was introduced to Sofie in version 1.50. This gateway serves as a way for an external system to subscribe to publications which are designed to be simpler than the ones we publish over DDP. These publications are intended to be used by external systems which need a 'stable' API and to not have too much knowledge about the inner workings of Sofie. See Api Stability for more details.