Pattern Adoption Stories from Readers: Dutch Government & Energy Sector (2/4)

On pages like this, we collect contributions from readers of our patterns.

Part 1 Part 3

This pattern adoption story was contributed by Ton Donker, Ruben Haasjes and their readers group. It covers the 21 patterns presented as API Design Pattern of the Week on LinkedIn and Medium as well as a fourth slice with additional patterns. This page features part 2 of 4 (part 1, part 3, part 4).

State Transition Operation

Known Uses

In the Dutch Energy sector, the State Transition Operation pattern is commonly used in workflow management implementations that assign work tasks to contractors involved in the operational maintenance of the energy infrastructure network. Another known use is the sector-wide customer support ticket management platform. In this scenario, the API operations manage state in the ticket lifecycle algorithm.

Discussion Input

It’s good to distinguish between the terms “State Transition” and “State Transfer”. The “State Transition Operation” pattern takes state as input and modifies the provider-side application state, whereas “State Transfer Operation” does require state input in order to modify the state. To put the different terms in the functional notation as used in the book:

State Transition Operation: (in, S) -> (out, S')

State Transfer Operation: in -> (out, S')

“State Transition” corresponds nicely to Command Message in the Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf: messages with the intent to change the state of the system. They are actions to be performed that have side effects. “State Transfer” corresponds to Event Messages: events are things that have happened within a system. They are the result (not in the method/return sense) of an executed command. Consider in this context the term “Event Carried State Transfer”. In the book, “State Transfer Operation” is not used, but the term “State Creation Operation” is used instead. This is all explained in the “Notes from The Architecture and Modeling Learning Event”; we found the mapping of these terms to EIP very useful.

There was an interesting discussion on LinkedIn on the functional notation used in the book for the different operation responsibilities. How would a delete operation (do) be notated? Since it does affect the state and it is a variant of State Transition Operation (sto) it would advisable to use the same notation as for sto:

do: (in, S) -> (out, S')

However, what’s the added value of sending a response back when deleting? in HTTP APIs, one of the commonly used status codes reporting a successful deletion is a ‘204’ — which stands for: ‘No Content’ — which means ‘no response payload’ and since delete affects the state, perhaps the following notation would help for expressing the delete operation (do):

do: (in, S) -> S'

Metadata Element

Known Uses

The Metadata Element pattern is widely used in both Dutch government APIs and Dutch Energy sector APIs. HTTP header parameters like X-Request-ID, X-Correlation-ID and ETag (to mention a few) are applied very often.

In the Energy sector, we have introduced additional custom headers:

  • X-Object-ID (response): alternative to the header Location in case a complete URL should not be returned in the response. X-Object-ID contains the identification element of the created resource, which makes it a known use of the Id Element pattern.
  • X-Principal-Token (request): contains the unique identification of the calling party (person or system) in the context of a request. The exact interpretation of the field’s content is context-dependent; it could be, for example, an OAuth access token.

Both hold control and provenance metadata. Note that we have prefixed the parameters with X (although not recommended, see for instance here and here) but we still like this simple way of identifying custom headers.

The document “API-strategie” (in Dutch) describes a specific use of metadata information: Metadata Endpoints are provided. In an overall system, APIs do not exist in isolation. When APIs form a chain or collectively realize an interface, they contribute to ecosystem awareness. Therefore, APIs actively contribute to “context awareness” by providing metadata endpoints:

  • Metadata Endpoint for Application Properties: This endpoint provides detailed information about the application properties such as its name, release information (version number), description, and the standards used, along with their version numbers. This information will allow API consumers to understand the characteristics and specifications of the application they are interacting with (https://.../api/v1/app-info)
  • Metadata Endpoint for Application Status: This endpoint will provide real-time information about the application status. It could include a simple value like UP/DOWN to indicate whether the application is currently operational or not. This status information is crucial for API consumers to assess the availability and health of the application before making any requests. (https://.../api/v1/app-health)

By making these two Metadata Endpoints available, API providers ensure that consumers have access to essential information in a standardized manner, enabling them to interact with APIs more efficiently and with greater awareness of the underlying applications. Example: https://aandeslagmetdeomgevingswet.nl/publish/pages/168043/omgevingsdocumenten-ow-validatie-v2.json (see “Meta” resources).

In the Dutch government APIs, Metadata Elements like HTTP header Content-CRS and others are used in geospatial contexts. Via de ‘Content-CRS’ (CRS: Coordinate Reference System) header, the client can negotiate on the required content in the response. Request and response may be based on another coordinate reference system. This applies the HTTP-mechanism for content negotiation: https://docs.geostandaarden.nl/api/API-Strategie-mod-geo/#crs-negotiation. This is an example of provenance and control metadata.

Zalando uses the X-Flow-ID: a generic parameter to be passed through service APIs and events and written into log files and traces, comparable with the X-Request-ID and X-Correlation-ID we use in our sector. Again, we see known uses and realizations of provenance and control metadata.

A nice example of provenance metadata is the concept of an identification scheme. A domain object can be uniquely identified by an MRID (Master Resource IDentifier). To determine the (functional) source of a given MRID, each MRID attribute possesses a supplementary attribute called identificationScheme, which classifies the MRID using a URI. By using the identificationScheme and a conversion API, it is also possible to “reclassify” a given MRID, allowing access to additional resource information from another source (where the given resource corresponds to a different MRID value).

Since metadata is as important as the data itself, we observed additional patterns, not included in the book, such as Metadata Resources and even dedicated Metadata APIs. The concept of a metadata resource is used by various companies and standards:

And Metadata APIs:

Metadata Resources are special types of Information Holder Resources, and Metadata APIs can be positioned as Backend Integration APIs that expose such Metadata Resources. They often come as Solution-Internal APIs or Community APIs; the above two examples are Public APIs.

Discussion Input

Many RESTful API design books discuss metadata and the various methods of incorporating metadata into request and response messages. Apart from the standard protocol-level headers, metadata can also be included within the payload and reflected in status codes. While knowing how to transmit metadata is vital in API design, understanding the distinct types of metadata, their roles, purposes, and the kind of information each variant of metadata can offer is equally essential. So far, we have not seen an API design book in which metadata is qualified and categorized as is done in “Patterns for API Design”. According to the book, metadata falls into three main categories: control, aggregated and provenance metadata elements. In short this stands for respectively: “technical usage hints” (identifiers, filters, API Keys, hypermedia controls), “statistic and calculations” (page counters, total of results) and “description and provenance” (version number, owner, location information). Categorizing metadata this way adds a better understanding of the purpose and role of the specific metadata element and the information it contains, both for provider and consumer and especially in the case of custom protocol headers! Applying the Metadata Element pattern and its metadata qualifications will certainly add value to the API design and make API usage more accurate and consistent since metadata is as important as the data itself.

An important design discussion is where to store metadata: in protocol headers or in body payload? Example: Pagination in “plain” JSON is supported by HTTP headers with metadata (X-Pagination-Page, X-Pagination-Limit), whereas HAL incorporates the pagination parameters in the payload. The default use of standard headers and header extension is strongly dependent on the chosen protocol and platform. And in case of protocol bridging metadata can get lost in transition, not guaranteeing end-to-end quality of service. We are looking forward to read the pattern Context Representation.

When a Metadata Element is part of the message payload it can be prefixed by an underscore _ to denote its metadata functionality and to distinguish it from non-metadata elements.

REST API Design Rule Book by Mark Massé: Chapter 4 is dedicated to “Metadata Design”.

NL GOV profile for CloudEvents v0.3. Context Metadata. This document is a Dutch governmental profile of the CloudEvents specification for describing event data in common formats to provide interoperability across services, platforms and systems.

Id Element

Known Uses

  • The Id Element pattern is very often used both locally within the API (like: X-Request-ID, JWT-token, API Keys, ETag) as also globally (like: EAN code, article codes, ISO country codes).
  • As stated on the pattern website Terravis uses different identifiers for parcels called electronic parcel ID – this is comparable with like the SSCC serial shipping container code in logistics and transportation: https://www.gs1.nl/en/barcodes/more-types-of-codes/serial-shipping-container-codesscc/.
  • Zalando uses the X-Flow-ID als local identifier for tracking purposes: https://opensource.zalando.com/restful-api-guidelines/#233. The X-Flow-ID and each API specification must be provisioned with a globally unique and immutable API identifier: https://opensource.zalando.com/restful-api-guidelines/#215.
  • A specific use of Id Element is the usage of an URN in the “instance” part of RFC 7807 to find the corresponding entry in error logs: https://github.com/VNG-Realisatie/gemma-zaken/issues/1495 (in Dutch)
  • CloudEvents (Dutch NL profile) uniquely identify the event (the “id” is an required attribute may come in UUID or an ID counter maintained by the producer: https://vng-realisatie.github.io/NL-GOV-profile-for-CloudEvents/#context-attributes
  • Globally unique identifiers are commonly used in the Energy sector and the Dutch governmental domain. In the Dutch Energy sector we work with unique company and connection point identifiers. An EAN code, also known as the European Article Number or International Article Number, is a unique barcode standard used to identify parties and products globally managed by GS1, a global organization that sets standards for various industries. The EAN code has a fixed structure, consisting of a country code, company code, product code and check digit. For instance, Alpiq AG (Balance Responsible Party) in Switzerland has EAN code 7609999121203. The company EAN code has 13 digits, connection points have 18 digits.
  • Transaction log headers to make requests traceable throughout the network are: NLX is built as a core component of the Common Ground vision, which aims to convert or replace the current monolithic information systems of Dutch municipalities (and wider government) with a state of the art, API-first software landscape, fulfilling societal demand for automation, transparency and privacy: https://docs.nlx.io/reference-information/transaction-log-headers
  • Next to the above EAN code also the ISBN is a nice example of an global identifier in URN format and an element of the Published Language in Domain-Driven Design (DDD). There are different explanations of the concept of published language in the DDD literature. One of them is that it should be governed by a domain steering committee — which is an industry standard set by a consortium. (see https://www.youtube.com/watch?v=1eP7o2YpweA&t=1182s and a small thesaurus on Published Language: https://tdonker.github.io/domain-driven-design-links/publishedlanguage.html).

Discussion Input

  • The EAN code of an electricity or gas connection and the associated consumption data is classified personal data which can be directly or indirectly referenced to an individual natural person. Additionally, many laws recognize special categories of personal data — which are sensitive data that require extra protection. Examples of such data include the Citizen Service Number, or ‘Burger Service Number (BSN)’ in Dutch. and health or medical information. This means that security and privacy are important considerations when designing an API in order to avoid exposing confidential Id Elements like EANs and BSNs as path parameters in the URI.
  • According to the book, many “Id Elements typically also do not contain semantic type information” (page 281). This does not hold for the EAN and ISBN codes in the above examples.

Embedded Entity

Known Uses

This pattern is frequently used, especially in cases where the information is needed by consumers immediately. It reduces the number of requests and improves user experience.

Discussion Input

The design dilemma between providing Embedded Entities or Linked Information Holders is also known as the discussion between “eager loading” versus “lazy loading”. Eager loading loads all the data upfront in a single query (Embedded Entity), while lazy loading fetches related data on-demand when it’s needed (Linked Information Holder). Both approaches have their benefit: lazy loading can avoid loading unused data and eager loading minimizes the trips to the database.

In some cases, when the consumer needs all resources straight away, it is more efficient to fetch the nested resources all at once (eager loading). In a RESTful approach, this should not be the default behavior, we think. Therefore, applying this mechanism should be an explicit choice. The decision is thus left to the API consumers since they know at which moments they need additional information and precisely what that information is. In short, a resource can support “eager loading” (in addition to the default “lazy loading”), in which case the consumer can indicate that related data should be loaded along with the main resource.

The power of RESTful APIs truly comes to the forefront when the calling party can exercise significant control over the information they want to receive, and the delivering source provides information that assists the calling party in exercising that control.

In the Dutch Energy sector, we make use of the _expand parameter for enabling eager loading of Linked Information Holders (becoming an Embedded Entity when it is expanded, aka “resource expansion”) and the _fields parameter to select fields or combination of fields in the response explicitly (which can be seen as a realization of the Wish List pattern). Using these parameters, the client controls the information retrieval and can avoid unnecessary data transfer.

Now back to the design dilemma of embedding data elements vs providing links. When comparing the response on page 317 and page 323 in the book (customerProfile as Embedded Entity or Linked Information Holder) we would have decided for modelling customerProfile as an Embedded Entity and not as a Linked Information Holder. The reason for this design decision is that the customerProfile Embedded Entity can be seen as a composite attribute, clustered group of data elements or data group without an identifier (aka value objects), which is something else than an associated resource that can be identified by MRID Master Resource IDentifier (aka entities). In our design strategy, data groups are never accessible via a link; they only exist within the context of the owning API, whereas related resources can exist outside the context of the owning API, possibly by an alternative API. The _fields parameter can be applied to data elements or groups of data elements, and the _expand parameter to related resources referenced by links to reduce messages size, support eager loading, and allow clients to select the information they are interested in.

An example of a related resource provided in the book is the “policies” related resource collection (page 121):

curl -X GET http://localhost:8090/customers/rgpp0wkpec
{
  "customerId": "rgpp0wkpec",
  ...
  "_links": {
  ...
    "policies": {
      "href": "/customers/rgpp0wkpec/policies"
    }
  }
}

In Dutch Energy sector design strategy, resources are defined as top-level resource, subresource, associated resource, and resource reference. They can be referenced by links and/or be fully embedded in the response via the _expand parameter. A group of elements can be explicitly embedded in the response via the _fields parameter. So:

curl -X GET http://localhost:8080/customers/gktlipwhjr?_fields=customerProfile would only return the customerProfile data group:

{
  "customerProfile": {
    "firstname": "Robbie",
    "lastname": "Davenhall",
    "birthday": "1961-08-11T23:00:00.000+0000",
    "currentAddress": {
       "streetAddress": "1 Dunning Trail",
       "postalCode": "9511",
       "city": "Banga"
    }
  }
}

In summary it’s an essential design choice to decide between Embedded Entities and Linked Information Holders, each of them having their own mechanism in constructing the needed response shape.

Linked Information Holder

Known Uses

The Linked Information Holder pattern is mainly used in Dutch Government APIs and less in the Dutch Energy sector. Examples are below BAG APIs (BAG stands for Basisregistratie Adressen en Gebouwen and translates to Basic Registration of Addresses and Buildings in the Netherlands) that provide current data about addresses, addressable objects, and buildings. The links are formatted according to the HAL specification:

URI templating is supported to provide a standardized and efficient way to build flexible and dynamic URIs, promoting consistency and ease of use.

Discussion Input

When the Linked Information Holder references another API endpoint, the question arises who is managing this second endpoint? The Linked Information Holder approach raises organizational questions such as ownership, support, and evolution strategies, especially when they point to other APIs. The Dutch API Design rules has an important note on cross-domain links:

“The only exception when navigation controls are allowed to point to other APIs is when they share governance and security context. When doing so, the governing party must guarantee stability of links between the APIs, which means the target operation of navigational links may never change during the lifetime of (a major version of) the originating API. They must also share the same security context, otherwise clients have to exchange mixed credentials for different endpoints.”

Another discussion point is how to cope with versions numbers in the URI (assuming that the Version Identifier pattern has been chosen) and how to minimize breaking links. A major release of an API results in a new endpoint containing an new version number. An idea to promote compatibility might be to provide two links to a referenced endpoint, a versioned one and one without without a Version Identifier. The latter will be provided as Linked Information Holders and will always redirect to the most actual versioned endpoint.

Very well stated in the book is that “API clients and providers must agree on the meaning of the link relationships and be aware of coupling and side effects introduced” (page 322). This is often a underestimated aspect in applying the hypermedia concept.

Conditional Request

Known Uses

The combination of ETags with the header parameter If-None-Match is commonly used in Dutch Government APIs. Example: https://petstore.swagger.io/?url=https://raw.githubusercontent.com/VNG-Realisatie/zaken-api/master/src/openapi.yaml#/resultaten/resultaat_retrieve (see operation GET /resultaten/{uuid}). This example even illustrates the usage of multiple ETags. Furthermore in this API the operation HEAD is implemented to check whether the cache is stale or up to date. Using HEAD instead of GET in order to just fetch the most current ETag value is always an option to save bandwidth.

There are no examples of implementations of the If-Modified-Since in combination with ETag.

Conditional Requests are also used to support concurrency control in HTTP. The header If-Match and If-None-Match can be used.

Known uses are:

Discussion Input

Caching is an API capability related to confidentiality. In case of privacy sensitive information response caching is not allowed.

API Description

Known Uses

In the Dutch Public Sector, API documentation must be provided in the form of an OpenAPI definition document which conforms to the OpenAPI Specification (from v3 onwards). The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to RESTful APIs, which allows both humans and computers to discover and understand the capabilities of the service without access to source code or through network traffic inspection. An API is as good as the accompanying documentation. The documentation has to be easily findable, searchable, and publicly accessible. Most developers will first read the documentation before they start implementing. Hiding the technical documentation in PDF documents and/or behind a login creates a barrier for both developers and search engines.

To make the OAS document easy to find and to facilitate self-discovering clients, there should be one standard location where the OAS document is available for downloadr. Clients (such as Swagger UI or ReDoc) must be able to retrieve the document without having to authenticate. Furthermore, the CORS (Cross-Origin Resource Sharing) policy for this URI must allow external domains to read the documentation from a browser environment.

In the Dutch Energy Sector we think that documentation should never be a voluminous document. When extensive documentation seems necessary, the functionality is probably too complex…

Examples: In the API registry of the DSO developers portal (DSO: Environment and Planning Act of the Netherlands which is expected to come into force on January 1, 2024) we can see the following:

  • API OAS specification
  • Link to conceptual information model (semantics of the payload)
  • Link to request an API Key
  • Functional documentation of the API
  • Link to a dedicated forum to raise questions or provide feedback

Another example is https://developer.overheid.nl/apis/vng-brc. Again there is a link to the API OAS specification and a link to a dedicated GitHub repository. The latter provides business contexts and a standard way of raising issues. Also, (anonymous) test data can be obtained here, as well as a getting started guide.

Discussion Input

As illustrated by the above examples, it should be noted that the OAS files primarily focus on documenting API endpoints and their structure. Possible combinations of inputs and outputs, use cases, and other business-related aspects not directly related to the structure of the API endpoints are only minimally supported and should be added outside the OAS. Besides centralizing the API documentation on the developer portal, API store, or API catalog, the OpenAPI Specification provides functionality to include links to external docs in the Description part of the specification: https://spec.openapis.org/oas/v3.1.0#external-documentation-object.

Knowing the target audience helps in documenting the API. We distinguish at least two target audiences: the technical developer who works mainly with API description formalized by the OpenAPI Specification 3.0 (OAS 3) description standard. On the other hand, business analysts or other stakeholders are also interested in the business context and business-related aspects and solutions the APIs provide. In many cases (especially in generic designed public APIs), the syntactic interface description (the technical contract) must be supplemented with semantic specifications such as organizational information, business ownership, commercial aspects, non-functional requirements, support procedures, sequence diagrams, etc. This supplemental business-related piece of information and the OASs jointly form the API documentation (as we see it).

The API documentation can also document business drivers and supported business capabilities (if needed in domain terms). Amongst many advantages, it offers added value for technical developers to get acquainted with the business context, language, and requirements. It helps to answer the question: what high-value capability does the API offer the business, and what problem does it solve? These concerns are well-stated in the book: “The amount of documentation depends on the client provider relationship. Since APIs evolve, the documentation needs to be updated – which bears risk and costs”. Figure 9.4 (page 403) provides a great balanced template for an elaborate API description (another takeaway! 😊). However, it is not that easy to decide between minimal and elaborate API descriptions in our experience.

Regardless of whether you decide to keep an API private or release it to the general public, documentation is incredibly important and often neglected. Documenting software is never a popular task, but it is paramount in the API design-first approach moving toward an effective API design before writing a line of production code.

References

Higginbotham, James. 2021. Principles of Web API Design: Delivering Value with APIs and Microservices. Addison-Wesley.

Back to homepage