Pattern Adoption Stories from Readers: Dutch Government & Energy Sector

On pages like this, we collect contributions from readers of our patterns.

Book Resources

This pattern adoption story was contributed by Ton Donker, Ruben Haasjes and their readers group. At present, it covers the 21 patterns presented as API Design Pattern of the Week on LinkedIn and Medium.

Wish List

Known Uses

The Wish List pattern is widely used in Dutch government APIs. The guidelines for use of this pattern are available online in Section 9 (“Customization”) of “API Designrules Extensions”.

Government APIs are available via a public developer portal. Please find below some examples of Dutch government APIs that make use of the fields parameter for custom representation:

  1. https://developer.overheid.nl/apis/kadaster-bag-current/specificatie/production
  2. https://developer.overheid.nl/apis/logius-cor/specificatie/production

In the Dutch Energy sector I am currently working in, we are about to design APIs that utilize the combination of the fields parameter with the expand parameter. In HAL-formatted responses (responses with links to associated resources) the consumer can retrieve subfields of the associated resource by expanding the returned HAL link. Design rules are available in the Dutch document “API-strategie”. There is no concrete API yet.

Discussion points:

  • To distinguish between “real” query parameters and the more “steering” or “meta” parameters like expand, fields, limit and page, we – the Dutch Energy sector API Working Group – advocate the usage of an underscore prefix, so _fields, _expand, _limit and _page to prevent misinterpretation of the function of the query parameters.
  • The Wish List works in “optima forma” or comes to his right if all the response fields are optional. If there are any required fields in the response, the question arises if these fields should be returned in case they are not requested in the Wish List;
  • The implementation in provider service layers of the Wish List is expensive - sometimes these costs are simply too high and is there no business case to justify the investment.

Some other sources that mention the Wish List pattern:

More comments about the pattern can be found on Linked In.

Pagination

Known Uses

  1. Dutch governmental and Dutch Energy APIs make widely use of the page-based pagination variant. See for instance Section 14 of “API Designrules Extensions”.
  2. Nordic API distinguishes five types of pagination methods, see “Everything You Need to Know About API Pagination”.

We see a current tendency in the industry to favor the use of the cursor-paged pagination variant. Examples include (just to mention a few):

In the Dutch Energy sector we are analyzing this tendency.

The pattern was discussed rather intensively in week 2 of our “API Design Pattern of the Week” series on Linked In.

A very tricky pattern, each variant has its pros and cons and there is no ‘one-size-fits-all solution’. Real-time data and large datasets?: cursor-based pagination… Optimal DX and static data?: offset-based pagination…, just to mention a few. More decision drivers and design concerns in the book.

Error Report

Known Uses

  1. RFC7807 is adopted by Dutch governmental API design rules, see Section 17 of “API Designrules Extensions”;
  2. Same in the Dutch Energy sector, which declared the RFC7807 members type, title and status as required;
  3. Zalando http://opensource.zalando.com/restful-api-guidelines/#176 propose relative URI references in type field;
  4. Swedbank pay https://developer.swedbankpay.com/introduction.html#problems.

Discussion points:

  1. Should error handling include all possible errors or should it simply stop processing after the first error? The second scenario is how most APIs currently work since it less implementation effort. In case of Request Bundle (p292 of the book) first scenario is advisory;
  2. In the book (p290) it is stated: “Timestamps are another common information in Error Reports too”. Good point! However timestamp is not a member of RFC 7807 and seems to be absent in other alternative error response formats;
  3. Error reports should enable the receiver to be self-serviced. A talk given by Asbjørn Ulsberg is on this: https://www.youtube.com/watch?v=ZWIUn5BHDBc.

Also see comments on the pattern on Linked In. Feel free to join the discussion!

API Key

I think the pattern is very well explained in the book.

Known Uses

The Section 3.5 in “API Designrules Extensions” details the authentication and authorization protocol in the Dutch governmental sector (in English).

Discussion Input

In the case of authentication/authorization errors, the details of the error report should be carefully considered and minimized not to unveil any provider-side implementation security and privacy details. So the pattern API Key requires special implementation of Error Report; in that case, an API Key impacts the contents and message representation design of Error Report. This is all very well explained in the book, and it is also well-stated that an API Key is a “lightweight alternative to a full-fledged authentication protocol.” Despite all the concerns and ‘weaknesses’ one can argue that API Keys are a way to identify your consumers and build client communities to announce new releases and improve the client developer experience.

Processing Resource

The notion of Processing Resource as an endpoint role offering an activity-oriented API endpoint automatically brings us to the discussion how to define transactions as (REST) resources in the URI endpoint of the HTTP API. Should the URL be verb-free? This is still a hot debate in the REST API design community (so is the discussion between REST vs RPC). Some will argue that using verbs in data-oriented API endpoints will enrich the semantics of the API, others see it as an anti-pattern. Further inspiration on this subject: https://tyk.io/blog/rest-never-crud/ by James Higginbotham. This valuable link is of course provided in the book (amongst many others)!

Known Uses

n/a

The notion of activity-oriented API roles and processing resources endpoints is in the original literature very often associated with the controller resource archetype. Examples:

  • RESTful Web Service Cookbook (Allamaraju 2010): In order to abstract complex business operations (including transactions), servers may need to employ controller resources to make changes to other resources (e.g. Recipe 2.6);
  • REST API Design Rulebook (Massé 2012): A controller resource models a procedural concept. Controller resources are like executable functions, with parameters and return values; inputs and outputs;
  • https://restfulapi.net/resource-naming/: Use verb to denote controller archetype;

Informative link and resource that talk about RPC and REST (understanding the difference might help in understanding the Processing Resource pattern:

With regard to the API endpoints must be verb-free debate it’s worthwhile to mention:

This is of course a simple fragment of what is being offered by the web community, hopefully it adds some inspiration and ideas!

Information Holder Resource

Perhaps one of the most frequently used design patterns, especially in combination with Retrieval Operations Good to read that joint use of Processing Resource and Information Holder Resource is not recommended for microservices architectures (responsibility separation). And tagging the Information Holder Resource as one of the subtypes certainly adds semantics and expressiveness to the contract, establishing a ‘ubiquitous language’ used in API design… 👍

Known Uses

As already stated in the book, this pattern is widely used in may public Web APIs.

Known uses:

There are many other known uses of this pattern, so above links just mentions a few. It’s a typical pattern used in governmental APIs.

Data Transfer Resource

Known Uses

In the Dutch Energy sector we have EDSN (Energy Data Services Netherlands) as central data platform for market-facilitating. Cooperation and communication between market parties (suppliers, grid operators, metering companies) is facilitated via the central infrastructure of EDSN. There are use cases in which central data needs to be shared from the source with the decentral market party sinks (for instance for reporting purposes). This is achieved via de Data Transfer Resource Pattern, technically implemented by the S3 cloud storage protocol (the so-called S3 buckets). Such cloud storage is a technique that allows for the exchange of large data sets, low implementation costs, availability, durability, scalability, decoupling and it has a broad adoption in the industry (most parties are capable of connecting to cloud storage).

The pattern variant currently in place is Relay Resource: Every market party sink has a unique bucket where only the required semi-finished data products are delivered to. Housekeeping rules (like retention periods) are mutually agreed upon, and based on business processes. Replay and deletion are responsibilities of the data sink. The shared resource address is agreed upon in advance. Future plans are to optimize the process and to use push notifications (event driven) to inform clients proactively in order to enable faster processing of data and discoverability (which links nicely to the Link Lookup Resource Pattern as described in the book 😊!)

Further known uses from public cloud providers are:

Discussion Input

There are of course security and privacy risks when enterprises store and share files via a Web-flavored shared repository (like in the cloud). However, there are ways to avoid these risks, and the advantage of having all your data in one convenient place that can be accessed at any time from any place is worth investigating as long as the implementation is compliant with government regulations, security and privacy rules, and other legal obligations.

State Transition Operation

Known Uses

In the Dutch Energy sector, the State Transition Operation pattern is commonly used in workflow management implementations that assign work tasks to contractors involved in the operational maintenance of the energy infrastructure network. Another known use is the sector-wide customer support ticket management platform. In this scenario, the API operations manage state in the ticket lifecycle algorithm.

Discussion Input

It’s good to distinguish between the terms “State Transition” and “State Transfer”. The “State Transition Operation” pattern takes state as input and modifies the provider-side application state, whereas “State Transfer Operation” does require state input in order to modify the state. To put the different terms in the functional notation as used in the book:

State Transition Operation: (in, S) -> (out, S')

State Transfer Operation: in -> (out, S')

“State Transition” corresponds nicely to Command Message in the Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf: messages with the intent to change the state of the system. They are actions to be performed that have side effects. “State Transfer” corresponds to Event Messages: events are things that have happened within a system. They are the result (not in the method/return sense) of an executed command. Consider in this context the term “Event Carried State Transfer”. In the book, “State Transfer Operation” is not used, but the term “State Creation Operation” is used instead. This is all explained in the “Notes from The Architecture and Modeling Learning Event”; we found the mapping of these terms to EIP very useful.

There was an interesting discussion on LinkedIn on the functional notation used in the book for the different operation responsibilities. How would a delete operation (do) be notated? Since it does affect the state and it is a variant of State Transition Operation (sto) it would advisable to use the same notation as for sto:

do: (in, S) -> (out, S')

However, what’s the added value of sending a response back when deleting? in HTTP APIs, one of the commonly used status codes reporting a successful deletion is a ‘204’ — which stands for: ‘No Content’ — which means ‘no response payload’ and since delete affects the state, perhaps the following notation would help for expressing the delete operation (do):

do: (in, S) -> S'

Metadata Element

Known Uses

The Metadata Element pattern is widely used in both Dutch government APIs and Dutch Energy sector APIs. HTTP header parameters like X-Request-ID, X-Correlation-ID and ETag (to mention a few) are applied very often.

In the Energy sector, we have introduced additional custom headers:

  • X-Object-ID (response): alternative to the header Location in case a complete URL should not be returned in the response. X-Object-ID contains the identification element of the created resource, which makes it a known use of the Id Element pattern.
  • X-Principal-Token (request): contains the unique identification of the calling party (person or system) in the context of a request. The exact interpretation of the field’s content is context-dependent; it could be, for example, an OAuth access token.

Both hold control and provenance metadata. Note that we have prefixed the parameters with X (although not recommended, see for instance here and here) but we still like this simple way of identifying custom headers.

The document “API-strategie” (in Dutch) describes a specific use of metadata information: Metadata Endpoints are provided. In an overall system, APIs do not exist in isolation. When APIs form a chain or collectively realize an interface, they contribute to ecosystem awareness. Therefore, APIs actively contribute to “context awareness” by providing metadata endpoints:

  • Metadata Endpoint for Application Properties: This endpoint provides detailed information about the application properties such as its name, release information (version number), description, and the standards used, along with their version numbers. This information will allow API consumers to understand the characteristics and specifications of the application they are interacting with (https://.../api/v1/app-info)
  • Metadata Endpoint for Application Status: This endpoint will provide real-time information about the application status. It could include a simple value like UP/DOWN to indicate whether the application is currently operational or not. This status information is crucial for API consumers to assess the availability and health of the application before making any requests. (https://.../api/v1/app-health)

By making these two Metadata Endpoints available, API providers ensure that consumers have access to essential information in a standardized manner, enabling them to interact with APIs more efficiently and with greater awareness of the underlying applications. Example: https://aandeslagmetdeomgevingswet.nl/publish/pages/168043/omgevingsdocumenten-ow-validatie-v2.json (see “Meta” resources).

In the Dutch government APIs, Metadata Elements like HTTP header Content-CRS and others are used in geospatial contexts. Via de ‘Content-CRS’ (CRS: Coordinate Reference System) header, the client can negotiate on the required content in the response. Request and response may be based on another coordinate reference system. This applies the HTTP-mechanism for content negotiation: https://docs.geostandaarden.nl/api/API-Strategie-mod-geo/#crs-negotiation. This is an example of provenance and control metadata.

Zalando uses the X-Flow-ID: a generic parameter to be passed through service APIs and events and written into log files and traces, comparable with the X-Request-ID and X-Correlation-ID we use in our sector. Again, we see known uses and realizations of provenance and control metadata.

A nice example of provenance metadata is the concept of an identification scheme. A domain object can be uniquely identified by an MRID (Master Resource IDentifier). To determine the (functional) source of a given MRID, each MRID attribute possesses a supplementary attribute called identificationScheme, which classifies the MRID using a URI. By using the identificationScheme and a conversion API, it is also possible to “reclassify” a given MRID, allowing access to additional resource information from another source (where the given resource corresponds to a different MRID value).

Since metadata is as important as the data itself, we observed additional patterns, not included in the book, such as Metadata Resources and even dedicated Metadata APIs. The concept of a metadata resource is used by various companies and standards:

And Metadata APIs:

Metadata Resources are special types of Information Holder Resources, and Metadata APIs can be positioned as Backend Integration APIs that expose such Metadata Resources. They often come as Solution-Internal APIs or Community APIs; the above two examples are Public APIs.

Discussion Input

Many RESTful API design books discuss metadata and the various methods of incorporating metadata into request and response messages. Apart from the standard protocol-level headers, metadata can also be included within the payload and reflected in status codes. While knowing how to transmit metadata is vital in API design, understanding the distinct types of metadata, their roles, purposes, and the kind of information each variant of metadata can offer is equally essential. So far, we have not seen an API design book in which metadata is qualified and categorized as is done in “Patterns for API Design”. According to the book, metadata falls into three main categories: control, aggregated and provenance metadata elements. In short this stands for respectively: “technical usage hints” (identifiers, filters, API Keys, hypermedia controls), “statistic and calculations” (page counters, total of results) and “description and provenance” (version number, owner, location information). Categorizing metadata this way adds a better understanding of the purpose and role of the specific metadata element and the information it contains, both for provider and consumer and especially in the case of custom protocol headers! Applying the Metadata Element pattern and its metadata qualifications will certainly add value to the API design and make API usage more accurate and consistent since metadata is as important as the data itself.

An important design discussion is where to store metadata: in protocol headers or in body payload? Example: Pagination in “plain” JSON is supported by HTTP headers with metadata (X-Pagination-Page, X-Pagination-Limit), whereas HAL incorporates the pagination parameters in the payload. The default use of standard headers and header extension is strongly dependent on the chosen protocol and platform. And in case of protocol bridging metadata can get lost in transition, not guaranteeing end-to-end quality of service. We are looking forward to read the pattern Context Representation.

When a Metadata Element is part of the message payload it can be prefixed by an underscore _ to denote its metadata functionality and to distinguish it from non-metadata elements.

REST API Design Rule Book by Mark Massé: Chapter 4 is dedicated to “Metadata Design”.

NL GOV profile for CloudEvents v0.3. Context Metadata. This document is a Dutch governmental profile of the CloudEvents specification for describing event data in common formats to provide interoperability across services, platforms and systems.

Id Element

Known Uses

  • The Id Element pattern is very often used both locally within the API (like: X-Request-ID, JWT-token, API Keys, ETag) as also globally (like: EAN code, article codes, ISO country codes).
  • As stated on the pattern website Terravis uses different identifiers for parcels called electronic parcel ID – this is comparable with like the SSCC serial shipping container code in logistics and transportation: https://www.gs1.nl/en/barcodes/more-types-of-codes/serial-shipping-container-codesscc/.
  • Zalando uses the X-Flow-ID als local identifier for tracking purposes: https://opensource.zalando.com/restful-api-guidelines/#233. The X-Flow-ID and each API specification must be provisioned with a globally unique and immutable API identifier: https://opensource.zalando.com/restful-api-guidelines/#215.
  • A specific use of Id Element is the usage of an URN in the “instance” part of RFC 7807 to find the corresponding entry in error logs: https://github.com/VNG-Realisatie/gemma-zaken/issues/1495 (in Dutch)
  • CloudEvents (Dutch NL profile) uniquely identify the event (the “id” is an required attribute may come in UUID or an ID counter maintained by the producer: https://vng-realisatie.github.io/NL-GOV-profile-for-CloudEvents/#context-attributes
  • Globally unique identifiers are commonly used in the Energy sector and the Dutch governmental domain. In the Dutch Energy sector we work with unique company and connection point identifiers. An EAN code, also known as the European Article Number or International Article Number, is a unique barcode standard used to identify parties and products globally managed by GS1, a global organization that sets standards for various industries. The EAN code has a fixed structure, consisting of a country code, company code, product code and check digit. For instance, Alpiq AG (Balance Responsible Party) in Switzerland has EAN code 7609999121203. The company EAN code has 13 digits, connection points have 18 digits.
  • Transaction log headers to make requests traceable throughout the network are: NLX is built as a core component of the Common Ground vision, which aims to convert or replace the current monolithic information systems of Dutch municipalities (and wider government) with a state of the art, API-first software landscape, fulfilling societal demand for automation, transparency and privacy: https://docs.nlx.io/reference-information/transaction-log-headers
  • Next to the above EAN code also the ISBN is a nice example of an global identifier in URN format and an element of the Published Language in Domain-Driven Design (DDD). There are different explanations of the concept of published language in the DDD literature. One of them is that it should be governed by a domain steering committee — which is an industry standard set by a consortium. (see https://www.youtube.com/watch?v=1eP7o2YpweA&t=1182s and a small thesaurus on Published Language: https://tdonker.github.io/domain-driven-design-links/publishedlanguage.html).

Discussion Input

  • The EAN code of an electricity or gas connection and the associated consumption data is classified personal data which can be directly or indirectly referenced to an individual natural person. Additionally, many laws recognize special categories of personal data — which are sensitive data that require extra protection. Examples of such data include the Citizen Service Number, or ‘Burger Service Number (BSN)’ in Dutch. and health or medical information. This means that security and privacy are important considerations when designing an API in order to avoid exposing confidential Id Elements like EANs and BSNs as path parameters in the URI.
  • According to the book, many “Id Elements typically also do not contain semantic type information” (page 281). This does not hold for the EAN and ISBN codes in the above examples.

Embedded Entity

Known Uses

This pattern is frequently used, especially in cases where the information is needed by consumers immediately. It reduces the number of requests and improves user experience.

Discussion Input

The design dilemma between providing Embedded Entities or Linked Information Holders is also known as the discussion between “eager loading” versus “lazy loading”. Eager loading loads all the data upfront in a single query (Embedded Entity), while lazy loading fetches related data on-demand when it’s needed (Linked Information Holder). Both approaches have their benefit: lazy loading can avoid loading unused data and eager loading minimizes the trips to the database.

In some cases, when the consumer needs all resources straight away, it is more efficient to fetch the nested resources all at once (eager loading). In a RESTful approach, this should not be the default behavior, we think. Therefore, applying this mechanism should be an explicit choice. The decision is thus left to the API consumers since they know at which moments they need additional information and precisely what that information is. In short, a resource can support “eager loading” (in addition to the default “lazy loading”), in which case the consumer can indicate that related data should be loaded along with the main resource.

The power of RESTful APIs truly comes to the forefront when the calling party can exercise significant control over the information they want to receive, and the delivering source provides information that assists the calling party in exercising that control.

In the Dutch Energy sector, we make use of the _expand parameter for enabling eager loading of Linked Information Holders (becoming an Embedded Entity when it is expanded, aka “resource expansion”) and the _fields parameter to select fields or combination of fields in the response explicitly (which can be seen as a realization of the Wish List pattern). Using these parameters, the client controls the information retrieval and can avoid unnecessary data transfer.

Now back to the design dilemma of embedding data elements vs providing links. When comparing the response on page 317 and page 323 in the book (customerProfile as Embedded Entity or Linked Information Holder) we would have decided for modelling customerProfile as an Embedded Entity and not as a Linked Information Holder. The reason for this design decision is that the customerProfile Embedded Entity can be seen as a composite attribute, clustered group of data elements or data group without an identifier (aka value objects), which is something else than an associated resource that can be identified by MRID Master Resource IDentifier (aka entities). In our design strategy, data groups are never accessible via a link; they only exist within the context of the owning API, whereas related resources can exist outside the context of the owning API, possibly by an alternative API. The _fields parameter can be applied to data elements or groups of data elements, and the _expand parameter to related resources referenced by links to reduce messages size, support eager loading, and allow clients to select the information they are interested in.

An example of a related resource provided in the book is the “policies” related resource collection (page 121):

curl -X GET http://localhost:8090/customers/rgpp0wkpec
{
  "customerId": "rgpp0wkpec",
  ...
  "_links": {
  ...
    "policies": {
      "href": "/customers/rgpp0wkpec/policies"
    }
  }
}

In Dutch Energy sector design strategy, resources are defined as top-level resource, subresource, associated resource, and resource reference. They can be referenced by links and/or be fully embedded in the response via the _expand parameter. A group of elements can be explicitly embedded in the response via the _fields parameter. So:

curl -X GET http://localhost:8080/customers/gktlipwhjr?_fields=customerProfile would only return the customerProfile data group:

{
  "customerProfile": {
    "firstname": "Robbie",
    "lastname": "Davenhall",
    "birthday": "1961-08-11T23:00:00.000+0000",
    "currentAddress": {
       "streetAddress": "1 Dunning Trail",
       "postalCode": "9511",
       "city": "Banga"
    }
  }
}

In summary it’s an essential design choice to decide between Embedded Entities and Linked Information Holders, each of them having their own mechanism in constructing the needed response shape.

Linked Information Holder

Known Uses

The Linked Information Holder pattern is mainly used in Dutch Government APIs and less in the Dutch Energy sector. Examples are below BAG APIs (BAG stands for Basisregistratie Adressen en Gebouwen and translates to Basic Registration of Addresses and Buildings in the Netherlands) that provide current data about addresses, addressable objects, and buildings. The links are formatted according to the HAL specification:

URI templating is supported to provide a standardized and efficient way to build flexible and dynamic URIs, promoting consistency and ease of use.

Discussion Input

When the Linked Information Holder references another API endpoint, the question arises who is managing this second endpoint? The Linked Information Holder approach raises organizational questions such as ownership, support, and evolution strategies, especially when they point to other APIs. The Dutch API Design rules has an important note on cross-domain links:

“The only exception when navigation controls are allowed to point to other APIs is when they share governance and security context. When doing so, the governing party must guarantee stability of links between the APIs, which means the target operation of navigational links may never change during the lifetime of (a major version of) the originating API. They must also share the same security context, otherwise clients have to exchange mixed credentials for different endpoints.”

Another discussion point is how to cope with versions numbers in the URI (assuming that the Version Identifier pattern has been chosen) and how to minimize breaking links. A major release of an API results in a new endpoint containing an new version number. An idea to promote compatibility might be to provide two links to a referenced endpoint, a versioned one and one without without a Version Identifier. The latter will be provided as Linked Information Holders and will always redirect to the most actual versioned endpoint.

Very well stated in the book is that “API clients and providers must agree on the meaning of the link relationships and be aware of coupling and side effects introduced” (page 322). This is often a underestimated aspect in applying the hypermedia concept.

Conditional Request

Known Uses

The combination of ETags with the header parameter If-None-Match is commonly used in Dutch Government APIs. Example: https://petstore.swagger.io/?url=https://raw.githubusercontent.com/VNG-Realisatie/zaken-api/master/src/openapi.yaml#/resultaten/resultaat_retrieve (see operation GET /resultaten/{uuid}). This example even illustrates the usage of multiple ETags. Furthermore in this API the operation HEAD is implemented to check whether the cache is stale or up to date. Using HEAD instead of GET in order to just fetch the most current ETag value is always an option to save bandwidth.

There are no examples of implementations of the If-Modified-Since in combination with ETag.

Conditional Requests are also used to support concurrency control in HTTP. The header If-Match and If-None-Match can be used.

Known uses are:

Discussion Input

Caching is an API capability related to confidentiality. In case of privacy sensitive information response caching is not allowed.

API Description

Known Uses

In the Dutch Public Sector, API documentation must be provided in the form of an OpenAPI definition document which conforms to the OpenAPI Specification (from v3 onwards). The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to RESTful APIs, which allows both humans and computers to discover and understand the capabilities of the service without access to source code or through network traffic inspection. An API is as good as the accompanying documentation. The documentation has to be easily findable, searchable, and publicly accessible. Most developers will first read the documentation before they start implementing. Hiding the technical documentation in PDF documents and/or behind a login creates a barrier for both developers and search engines.

To make the OAS document easy to find and to facilitate self-discovering clients, there should be one standard location where the OAS document is available for downloadr. Clients (such as Swagger UI or ReDoc) must be able to retrieve the document without having to authenticate. Furthermore, the CORS (Cross-Origin Resource Sharing) policy for this URI must allow external domains to read the documentation from a browser environment.

In the Dutch Energy Sector we think that documentation should never be a voluminous document. When extensive documentation seems necessary, the functionality is probably too complex…

Examples: In the API registry of the DSO developers portal (DSO: Environment and Planning Act of the Netherlands which is expected to come into force on January 1, 2024) we can see the following:

  • API OAS specification
  • Link to conceptual information model (semantics of the payload)
  • Link to request an API Key
  • Functional documentation of the API
  • Link to a dedicated forum to raise questions or provide feedback

Another example is https://developer.overheid.nl/apis/vng-brc. Again there is a link to the API OAS specification and a link to a dedicated GitHub repository. The latter provides business contexts and a standard way of raising issues. Also, (anonymous) test data can be obtained here, as well as a getting started guide.

Discussion Input

As illustrated by the above examples, it should be noted that the OAS files primarily focus on documenting API endpoints and their structure. Possible combinations of inputs and outputs, use cases, and other business-related aspects not directly related to the structure of the API endpoints are only minimally supported and should be added outside the OAS. Besides centralizing the API documentation on the developer portal, API store, or API catalog, the OpenAPI Specification provides functionality to include links to external docs in the Description part of the specification: https://spec.openapis.org/oas/v3.1.0#external-documentation-object.

Knowing the target audience helps in documenting the API. We distinguish at least two target audiences: the technical developer who works mainly with API description formalized by the OpenAPI Specification 3.0 (OAS 3) description standard. On the other hand, business analysts or other stakeholders are also interested in the business context and business-related aspects and solutions the APIs provide. In many cases (especially in generic designed public APIs), the syntactic interface description (the technical contract) must be supplemented with semantic specifications such as organizational information, business ownership, commercial aspects, non-functional requirements, support procedures, sequence diagrams, etc. This supplemental business-related piece of information and the OASs jointly form the API documentation (as we see it).

The API documentation can also document business drivers and supported business capabilities (if needed in domain terms). Amongst many advantages, it offers added value for technical developers to get acquainted with the business context, language, and requirements. It helps to answer the question: what high-value capability does the API offer the business, and what problem does it solve? These concerns are well-stated in the book: “The amount of documentation depends on the client provider relationship. Since APIs evolve, the documentation needs to be updated – which bears risk and costs”. Figure 9.4 (page 403) provides a great balanced template for an elaborate API description (another takeaway! 😊). However, it is not that easy to decide between minimal and elaborate API descriptions in our experience.

Regardless of whether you decide to keep an API private or release it to the general public, documentation is incredibly important and often neglected. Documenting software is never a popular task, but it is paramount in the API design-first approach moving toward an effective API design before writing a line of production code.

Rate Limit

Known Uses

The Rate Limit pattern is commonly used in Dutch government APIs, particularly in open data government services. Rate Limits are a standard practice for these types of APIs and are technically implemented. However, client awareness and (how much of the Rate Limit is used?) is sometimes poorly managed. Often, one must settle for information about the Rate Limit on the developer portal or gateway, and limit headers in responses are not commonly implemented. The Rate Limit pattern is less frequently utilized in the Dutch energy sector because there are fewer customer public APIs that are internet-facing.

Some Dutch government known uses (to mention only a few):

  • The KNMI (Royal Netherlands Meteorological Institute) Data Platform provides access to open datasets maintained or generated by the KNMI on weather, climate and seismology. This platform grants access via registered API Keys and anonymous keys. Anonymous keys provide unregistered access to the Open Data API. To ensure fair usage and to be able to control the operational costs of KNMI Data Platform, the number of API calls is limited with different limits for registered and anonymous keys. Note that anonymous users share these limits with all other active users of the anonymous API key. On the portal, the Rate Limit per registration policy is presented:
  • An example is the BAG API (BAG stands for Basisregistratie Adressen en Gebouwen and translates to Basic Registration of Addresses and Buildings in the Netherlands): This API uses two Rate Limit scenarios:
    • Kadaster-RateLimit-DayLimit: allowed amount of requests per day;
    • RateLimit-Limit: allowed amount of requests per second; In this example, limit headers are typically returned with every request, not exclusively with a 429 HTTP status code.
  • In the DSO API register (DSO: Environment and Planning Act of the Netherlands), which is expected to come into force on January 1, 2024, we can see an overview of APIs with allowed throttling limits (again no limit headers): https://aandeslagmetdeomgevingswet.nl/ontwikkelaarsportaal/api-register/beschikbare-api-services/.
  • In the KVK (Companies listed in the Business Register) we see a combination of Rate Limit and the Pricing Plan pattern: https://developers.kvk.nl/apis/zoeken.
  • OTTO Market API Developer’s Guide gives an example of Rate Limits per endpoint or partner id.

Discussion Input

Currently, there is no official, agreed-upon standard policy for usage in Dutch governmental API guidelines (as far as we know). Most APIs operate on a fair-use basis (very often a max of 100 requests per second) but without a proper description of what that entails. Usage limits are defined technically but not communicated clearly either on the portal or by headers in the API description itself.

The GitHub example presented in the book (page 414) is also used by Zalando and Otto, with the same custom HTTP headers. Besides the X-RateLimit scenario, Zalando also opts for the Retry-After scenario: a header indicating how long the client ought to wait before making a follow-up request. The reason to allow both approaches is that APIs can have different needs. For Zalando and Otto, headers MUST be used in combination with code 429. Also here: Rate Limit headers are generally returned on every request and not just on a 429:

Rate Limit is an important pattern according to OWASP “API4:2023 Unrestricted Resource Consumption”: https://owasp.org/API-Security/editions/2023/en/0xa4-unrestricted-resource-consumption/.

A nice discussion point is the question: what is the appropriate HTTP error code when a data-bound Rate Limit is violated? The 429 is used when there are too many requests, but that’s a different limit than the data-bound Rate Limit that includes response data volumes in the definition. How should one respond when the limit on data volume is surpassed? The recommended approach remains to utilize the 429 “Too Many Requests” error, supplemented with an RFC 7807 detailed explanation.

OpenAI has very good documentation on their API Rate Limit policy: https://platform.openai.com/docs/guides/rate-limits?context=tier-free.

Difference between Rate Limit and Throttling: Throttling is the action of controlling the speed or frequency of accesses, while rate limiting specifically sets a cap on the number of accesses within a defined timeframe. Throttling can involve various strategies, including rate limiting, but rate limiting is a specific form of throttling. (throttling: server side, Rate Limit client side). See https://nordicapis.com/api-rate-limiting-vs-api-throttling-how-are-they-different/.

It is interesting to see how the Rate Limit pattern relates to other patterns in the book. Since Rate Limit is an API-level pattern and the other ones listed below operate with message-Level scope (request messages in particular), it suggests a strong relationship between these two groups. Here is an attempt to overview the relations:

An important API security best practice is to use dynamic Rate Limits and set static Rate Limits selectively. Rate-limiting mechanisms should be much more dynamic, granular, and based on actual consumption patterns. Setting blanket static Rate Limits can result in impeded application functionality that directly impacts the organization’s business, not to mention attackers will throttle their attempts to evade those limits (source: https://salt.security/blog/api-security-best-practices).

Currently, there is no standard way for servers to communicate quotas so that clients can throttle their requests to prevent errors. Not official yet, but currently in the make is the draft RFC: RateLimit header fields for HTTP, https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-ratelimit-headers-07.

Context Representation

Known Uses

The Context Representation pattern is a less obvious one. Context information such as authentication and authorization, is frequently managed by API gateway intermediaries using standardized and generic methods. This context information is often delegated to the underlying protocol (e.g., through HTTP headers), and an explicit Context Representation in the request or response payload is infrequent. Nevertheless, the book rightly emphasizes that, advocating for protocol independence, this pattern offers an alternative approach to relying solely on standard headers and the header extension capabilities of the networking protocol.

  • In the WSDL/SOAP era, this pattern was actively used within the Dutch Energy sector. A standard business document header (included as part of the XML payload) was defined, drawing from and inspired by industry standards such as UN/CEFACT Standard Business Document Header.
  • OAGIS ApplicationArea: This business document header contains context information like: CreationTimestamp, MessageID, CorrelationID, MessageVersion, SenderID, and so on.

Another use case is NL GOV profile for CloudEvents. This is the Dutch government profile based on the CloudEvents 1.0.1 specification. The CloudEvents specification is geared more towards generic use, and in the Netherlands, we want to add a number of requirements for the Dutch situation with the goal to agree on how to use the CloudEvents specification. In the CloudEvents specification, context attributes are standardized metadata fields that provide essential information about the event itself. The core context attributes defined in CloudEvents include id, source, time, specversion, subject, datacontenttype and so on. These context attributes are typically bundled together as a structured set of metadata fields at the beginning of the event. They form a header-like structure that provides essential information about the event itself. However, they might not always be physically located at the very beginning of the event payload, depending on the transport or serialization format being used.

Discussion Input

This pattern requires additional attention, particularly as we pursue a more event-driven strategy in both the government and energy sectors. AsyncAPI is gaining more importance, and it brings integration camps (synchronous and asynchronous communication) together. New integration technologies and protocols will be introduced, ensuring the sustained relevance of this pattern in the future. To increase operability, allowing more API clients to connect, it can be beneficial to offer APIs in protocols and architectural styles other than REST/JSON. So a protocol agnostic approach is needed. The Context Representation pattern aims to help by normalizing payloads and metadata that producers and consumers can use to ensure logic does not break when the protocol changes. To be continued 😊!

Version Identifier

Known Uses

Versioning is a hotly debated topic in the API community and the book states correctly that the version strategies differ widely and are debated passionately. There are many questions and discussions floating around the internet about the correct way you should version your APIs. The HTTP header versioning (consumer gets to decide which version to request ) appeals to purist RESTafarians. In general, the most common approach is a combination of URI parameters and header criteria. However, in many cases, people tend to fixate blindly on the versioning issue. But the fact that there are so many perspectives on the topic of versioning probably indicates that there isn’t a single universal solution for this issue – there is no single ‘right way’. Perhaps the issue does not solely reside within the versioning strategy, but rather within the change strategy itself. The underlying problem is managing the impact of a change, and the appropriate version strategy can be applied based on this management. See Discussion Input below.

The book explains very well the various methods of implementing versioning to denote progress and maturity in API evolution. In addition to incorporating a Version Identifier in the URI, header, or payload, the more unconventional query-based versioning method is also utilized (however, we have never seen this in the wild).

Some known uses (to mention only a few):

Discussion Input

An interesting observation is that the zealous discussion about the version numbering of an API is a red herring: the version number is not the problem. Instead of a versioning strategy, you should have a change strategy. This observation basically propagates not doing versioning at all, or using it as a last resort since it causes long-term maintenance issues. This can be achieved by making use of hypermedia controls, by reconsidering new resources instead breaking existing ones by applying the correct bounded context boundaries and by developing a smart change strategy that embraces a compatible versioning strategy being always backwards compatible without needless duplication.

When discussing change strategy, we struggle (within Enexis) with balancing API complexity and client-side impact; to what extent should APIs be future-proof (forward-compatible)? As API designers, we often suggest common generic data structures even if a particular consumer does not request this. For instance, even if one consumer is only interested in the numerical value of a total amount we would suggest exposing this in the API as a value and a currency. Hence future-proofing the API for consumers that might require the currency. This example might introduce little extra complexity, but in practice, it’s hard to define guidelines on where future-proofing is helpful and when you are better off accepting a new API version in the future.

The point is that version numbering on a Web API “seems like a simple solution to a simple problem, while in fact it is a very naïve solution to a very complex problem” (quoting Asbjørn Ulsberg). This observation is supported by the following principles: Bertrand Meyer’s open-closed principle suggests that software entities should be open for extension but closed for modification. When applied to APIs, the implication is that you can augment your resources but not change them. You could do more to share the burden of change between API and client. Postel’s law, often referred to as the Robustness Principle, states that you should be liberal in what you accept and conservative in what you send. In terms of APIs, this implies a certain tolerance for consuming services.

More information on this topic can be found here:

Two in Production

Known Uses

The Two in Production pattern is actively employed in the API lifecycle management strategy of the Dutch government. It is interesting to mention that in the DSO API strategy, initially, the support was limited to a maximum of three versions of an API (“three in production”), comprising two major versions and one minor one. Both old and new versions (up to a maximum of three) of an API were concurrently offered for a limited transition period (up to one year), with two major versions available. Through the optional request header, one minor/patch version could be specified. This means that, for instance, in a (pre)production or integration environment, alongside, for example, v1 and v2, there was still the option to access one “older” or, alternatively, one “newer” minor/patch version of these APIs. This version could then be accessed through the request header. This approach was later discontinued due to significant operational complexity: running multiple minor/patch versions in parallel proved unnecessarily complicated. Furthermore, the use of an api-version request header conflicted with employing the version number in the URI. However, it was decided to include the api-version (full version number) in the response headers for every API call, since the URI only contains the major version. This information could then be used for logging, debugging or auditing purposes. In cases where an intermediate networking component returns an error response (e.g., a reverse proxy enforcing access policies), the version number may be omitted.

In the Dutch energy sector, we have opted for a maximum of three versions in production, allowing clients more time to upgrade and adapt within the managed evolution timeframe. This choice aims to strike a good balance between provider complexity and the pace of client adaptation.

Discussion Input

An important question is how many API versions in production should be offered in parallel. As stated very well in the book (page 391): the variant N in production gives clients more time and options to upgrade, but puts more maintenance effort and operational cost on the provider side. As always, it depends, and it is a trade-off between provider maintenance activities and pace of adapting to new major versions by clients. In any case, there should be a clear policy that defines the maximum number of major versions to be offered. The overall objective would be to keep the number of parallel major versions limited.

Public API

Known Uses

Public APIs are open and accessible to anyone who wants to use them. They are often free or have low fees. The goal of Public APIs is to make consumption easy and, very often, to attract as many consumers as possible. Obviously, there are many Public APIs offered in the Dutch Dutch Government Public sector, and they are available and discoverable via public developer portals like:

In these cases of open government data scenarios, very often the Retrieval Operation pattern is applied. In the Dutch Energy sector, there are fewer Public APIs. Some examples:

  • https://www.eancodeboek.nl/ With the help of this service, one can look up a ‘connection identification code’ (EAN code) for a connection to the electricity and gas networks in the Netherlands.
  • https://energieonderbrekingen.nl/api/v2/ Real-time information about gas and power outages. Collaboration of three Dutch grid operators: Enexis Netbeheer B.V., Liander N.V., and Stedin.

There are various ways to categorize and classify APIs. Commonly used is the classification into public, partner, and private. The Dutch government uses the classification internal/external and the further subdivision external/open and external/closed. The latter corresponds to the partner or community classification. See https://geonovum.github.io/KP-APIs/API-strategie-algemeen/Architectuur/#typologie-van-api-s.

It is very well stated in the book (page 49): ‘…the difference between a Public API in general and its Open API variant: a truly open API is a Public API without an API Key or other authentication means…’. See also: https://nordicapis.com/6-types-of-apis-open-public-partner-private-composite-unified/

Discussion Input

Figure 4.8 on page 142 illustrates a Public API in context. It shows a public API with three different consumers: 1) Web App 2) Mobile App 3) Third-Party systems. Within Enexis, we experience different API requirements from different consumers. For instance, frontend-oriented consumers might require form-like API data models that resemble web-forms, whereas Backend systems might require more detailed or generic information. These different requirements can be incorporated into a single Public API. However, the potential risk is creating cluttered APIs with redundant operations and data. How should we manage the requirements of different consumers? When is it justified to fulfill the requirements of different consumers in multiple Public APIs?

Retrieval Operation

Known Uses

Retrieval Operation is a very common pattern and is described in detail and elaborated upon in the book. Next to GraphQL, OData is a query-based API protocol standardized and managed by OASIS: https://www.odata.org/documentation/. Many companies, like SAP and Microsoft, offer OData APIs out of the box. OData provides advanced filer options, and it’s possible to define custom actions to execute robust queries.

Known uses in Dutch government:

  1. Valid: This is a moment in time on which the data returned is valid in reality.
  2. Available: This is a moment in time on which the data returned is available using the same API.
  3. Effective: This is a moment in time on which the data returned became legally effective.

The above time traveling example matches the Time-Bound report variant as described in the book.

Discussion Input

It is difficult to balance client-side flexibility and data model clarity. Some OData services expose large numbers of attributes, making it hard to understand the data model. Creating multiple Retrieval Operations can help scope the data model, but it remains challenging to determine optimal scopes for Retrieval Operations

Another point of discussion is the HTTP verb that should be used for Retrieval Operations It is possible to include a payload in a GET request, but it is not common practice nor recommended according to HTTP specifications. GET requests are primarily used to request data from a specified resource and should not have a payload in the usual sense. For complex queries, the POST operation as catch all method is used instead. But this is using a non-idempotent operation in an idempotent manner. That’s why recently, the QUERY method came into place as a new HTTP method for safe, idempotent query functionality with a request payload.

More information is available online:

Service Level Agreement (SLA)

Known Uses

Service Level Agreements (SLAs) are frequently incorporated into the terms and conditions of services, as denoted in the API Description itself (under “termsOfService” and contact details) within Dutch Government APIs. As legally binding documents, terms and conditions, along with SLAs for public APIs, are drafted, or at least reviewed and approved, by legal experts specializing in the field. However, they often manifest as unstructured and freeform texts, as outlined in the book on page 417.

In Dutch Energy sector the SLA is described in the Service Description Documentation (API user manual that provides service context information, usage examples, details on error handling, etc.) and the following service-level objectives are defined:

  • Response times
  • Availability
  • Uptime and peak load
  • Security
  • Confidentiality
  • Helpdesk availability (service window)

There are no examples where failing to meet an SLA could result in negative financial consequences or punitive penalties (as far as we know).

Examples:

Two examples in our domains are:

As correctly stated on api-patterns.org: “…many public cloud providers have explicit SLAs…”, the following can be added:

Discussion Input

In particular, when considering risk and accountability, notable differences exist between SLAs in the public and private sectors. Public sector SLAs typically encompass risk management associated with public interest, political implications, and adherence to public policy, necessitating elevated levels of accountability and transparency. Conversely, within the private sector, SLAs primarily center on managing financial risks, navigating market competition, and meeting the expectations of shareholders or investors.

ISO/IEC 19086-1 sets a framework to enhance the quality, reliability, and transparency of cloud Service Level Agreements aiming to improve the confidence and trust between cloud service providers and their customers: https://www.iso.org/standard/67545.html.

Wrap Up

Overall, I (Ton) found all patterns in the “API Design Pattern of the Week” series useful and have applied many of them (see above). The book “Patterns for API Design” is a must read for API designers, architects and developers. More comments in this LinkedIn article.

Thank you very much indeed, Ton! Chapter 10 of our book contains two more pattern adoption stories (Table of Content).

References

Higginbotham, James. 2021. Principles of Web API Design: Delivering Value with APIs and Microservices. Addison-Wesley.
Masse, M. 2011. REST API Design Rulebook: Designing Consistent RESTful Web Service Interfaces. O’Reilly Media. https://books.google.ch/books?id=eABpzyTcJNIC.

Back to homepage