A client that was using our products for sending out newsletters by email requested a feature that allows them to publish these newsletters in their own newsletter archive. To avoid creating a custom application it was decided to create a new generic solution useable by all our clients, where archived mailings can be retrieved using an API.

Storage

Sent mailings are stored in a MongoDB database. Here the mailing is split into two parts: the mailing data and the actual HTML document.

The mailing data contains information about the mailing itself including when it was sent, the sender name, which message was used and for which subscription it was sent. Additionally, extra information in the form of metadata is also stored with each mailing. This metadata is extracted from the HTML of the mailing and the API is designed to efficiently search for this metadata.

The non-personalized HTML of the mailing is stored separately and can be retrieved by using the archive mailing's unique ID.

Metadata

As mentioned earlier, the mailing is stored with additional metadata. These are key-values that can easily be added to a mailing by inserting these into the HTML itself. This is done according to the HTML Microdata specification. This is an extension to HTML which allows you to add structured data to an HTML document without it affecting the markup itself.

In order to read this Microdata a parser was created matching the full specification of HTML Microdata.

By using this specification you're not dependent on software doing this for you. However it also makes it easy for software to dynamically add metadata to the HTML, which is something we did by improving our editors so that users can easily add this metadata using a user interface without touching the HTML itself.

In practice the metadata can be used to, for example, add a newsletter name and language to the mailing. The client can then use the API to search for mailings with metadata matching this newsletter name and language.

API

In order to be able to retrieve the archived mailings, a RESTful ASP.NET Web API was developed. The feature has to be explicitly enabled for the client for them to be able to use it. The client can then call the API to search for mailings and use that to integrate the archive into their own website.

Authentication

The API is placed behind an API gateway called Kong. Among other things, this is used to manage the authentication of API users. This is done using OAuth 2.

Features

De API provides the following features.

Search mailings matching specific criteria

The client can retrieve a paged result of mailing data matching certain criteria. This criteria can for example specify the period in which the mailings are sent, which subscription was used and which metadata was added to the mailing. The result is a page of detailed mailing information, excluding the HTML as this is to be retrieved separately.

Retrieve a specific mailing by ID

When retrieving a page of mailings, the details described the ID of each archived mailing. This ID can be used to retrieve the same information but for a single mailing instead. Again, the HTML is not included in the result.

Retrieve the HTML of a mailing

The ID returned with the details of a mailing can be used to retrieve the HTML of this specific mailing. This is the non-personalized HTML, meaning no personal information of a recipient (such as a first or last name) is present in the HTML.

Search metadata keys matching specific criteria

Metadata is stored as key-value pairs. The same criteria that is used to search for mailings can be used to search for metadata keys only.

Search metadata values for a key and criteria

When you have knowledge of a metadata key, you can retrieve the values associated with this key. You can then also additionally specify mailing criteria to further filter the results. For example: if you have a metadata key name "newsletter-name", you can use this to retrieve all associated values to create a list of all available newsletter names present in the archive.

Example usage

The API could be used as follows:

1. The client adds the newsletter name as Microdata to the HTML;
2. The client uses the API to retrieve all available newsletter names by searching for the values of the metadata key;
3. The client uses the results to create an overview of all available newsletters;
4. A visitor clicks on one of the newsletters;
5. A paged of mailings for this newsletter is retrieved from the API;
6. The visitor chooses one of the mailings;
7. The HTML of the mailing is retrieved from the API and presented to the visitor.