Pattern Adoption Stories from Readers: Dutch Government & Energy Sector (3/4)

On pages like this, we collect contributions from readers of our patterns.

Part 2 Part 4

This pattern adoption story was contributed by Ton Donker, Ruben Haasjes and their readers group. It covers the 21 patterns presented as API Design Pattern of the Week on LinkedIn and Medium as well as other patterns. This page features part 3 of 4 (part 1, part 2, part 4).

Rate Limit

Known Uses

The Rate Limit pattern is commonly used in Dutch government APIs, particularly in open data government services. Rate Limits are a standard practice for these types of APIs and are technically implemented. However, client awareness and (how much of the Rate Limit is used?) is sometimes poorly managed. Often, one must settle for information about the Rate Limit on the developer portal or gateway, and limit headers in responses are not commonly implemented. The Rate Limit pattern is less frequently utilized in the Dutch energy sector because there are fewer customer public APIs that are internet-facing.

Some Dutch government known uses (to mention only a few):

  • The KNMI (Royal Netherlands Meteorological Institute) Data Platform provides access to open datasets maintained or generated by the KNMI on weather, climate and seismology. This platform grants access via registered API Keys and anonymous keys. Anonymous keys provide unregistered access to the Open Data API. To ensure fair usage and to be able to control the operational costs of KNMI Data Platform, the number of API calls is limited with different limits for registered and anonymous keys. Note that anonymous users share these limits with all other active users of the anonymous API key. On the portal, the Rate Limit per registration policy is presented:
  • An example is the BAG API (BAG stands for Basisregistratie Adressen en Gebouwen and translates to Basic Registration of Addresses and Buildings in the Netherlands): This API uses two Rate Limit scenarios:
    • Kadaster-RateLimit-DayLimit: allowed amount of requests per day;
    • RateLimit-Limit: allowed amount of requests per second; In this example, limit headers are typically returned with every request, not exclusively with a 429 HTTP status code.
  • In the DSO API register (DSO: Environment and Planning Act of the Netherlands), which is expected to come into force on January 1, 2024, we can see an overview of APIs with allowed throttling limits (again no limit headers): https://aandeslagmetdeomgevingswet.nl/ontwikkelaarsportaal/api-register/beschikbare-api-services/.
  • In the KVK (Companies listed in the Business Register) we see a combination of Rate Limit and the Pricing Plan pattern: https://developers.kvk.nl/apis/zoeken.
  • OTTO Market API Developer’s Guide gives an example of Rate Limits per endpoint or partner id.

Discussion Input

Currently, there is no official, agreed-upon standard policy for usage in Dutch governmental API guidelines (as far as we know). Most APIs operate on a fair-use basis (very often a max of 100 requests per second) but without a proper description of what that entails. Usage limits are defined technically but not communicated clearly either on the portal or by headers in the API description itself.

The GitHub example presented in the book (page 414) is also used by Zalando and Otto, with the same custom HTTP headers. Besides the X-RateLimit scenario, Zalando also opts for the Retry-After scenario: a header indicating how long the client ought to wait before making a follow-up request. The reason to allow both approaches is that APIs can have different needs. For Zalando and Otto, headers MUST be used in combination with code 429. Also here: Rate Limit headers are generally returned on every request and not just on a 429:

Rate Limit is an important pattern according to OWASP “API4:2023 Unrestricted Resource Consumption”: https://owasp.org/API-Security/editions/2023/en/0xa4-unrestricted-resource-consumption/.

A nice discussion point is the question: what is the appropriate HTTP error code when a data-bound Rate Limit is violated? The 429 is used when there are too many requests, but that’s a different limit than the data-bound Rate Limit that includes response data volumes in the definition. How should one respond when the limit on data volume is surpassed? The recommended approach remains to utilize the 429 “Too Many Requests” error, supplemented with an RFC 7807 detailed explanation.

OpenAI has very good documentation on their API Rate Limit policy: https://platform.openai.com/docs/guides/rate-limits?context=tier-free.

Difference between Rate Limit and Throttling: Throttling is the action of controlling the speed or frequency of accesses, while rate limiting specifically sets a cap on the number of accesses within a defined timeframe. Throttling can involve various strategies, including rate limiting, but rate limiting is a specific form of throttling. (throttling: server side, Rate Limit client side). See https://nordicapis.com/api-rate-limiting-vs-api-throttling-how-are-they-different/.

It is interesting to see how the Rate Limit pattern relates to other patterns in the book. Since Rate Limit is an API-level pattern and the other ones listed below operate with message-Level scope (request messages in particular), it suggests a strong relationship between these two groups. Here is an attempt to overview the relations:

An important API security best practice is to use dynamic Rate Limits and set static Rate Limits selectively. Rate-limiting mechanisms should be much more dynamic, granular, and based on actual consumption patterns. Setting blanket static Rate Limits can result in impeded application functionality that directly impacts the organization’s business, not to mention attackers will throttle their attempts to evade those limits (source: https://salt.security/blog/api-security-best-practices).

Currently, there is no standard way for servers to communicate quotas so that clients can throttle their requests to prevent errors. Not official yet, but currently in the make is the draft RFC: RateLimit header fields for HTTP, https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-ratelimit-headers-07.

Context Representation

Known Uses

The Context Representation pattern is a less obvious one. Context information such as authentication and authorization, is frequently managed by API gateway intermediaries using standardized and generic methods. This context information is often delegated to the underlying protocol (e.g., through HTTP headers), and an explicit Context Representation in the request or response payload is infrequent. Nevertheless, the book rightly emphasizes that, advocating for protocol independence, this pattern offers an alternative approach to relying solely on standard headers and the header extension capabilities of the networking protocol.

  • In the WSDL/SOAP era, this pattern was actively used within the Dutch Energy sector. A standard business document header (included as part of the XML payload) was defined, drawing from and inspired by industry standards such as UN/CEFACT Standard Business Document Header.
  • OAGIS ApplicationArea: This business document header contains context information like: CreationTimestamp, MessageID, CorrelationID, MessageVersion, SenderID, and so on.

Another use case is NL GOV profile for CloudEvents. This is the Dutch government profile based on the CloudEvents 1.0.1 specification. The CloudEvents specification is geared more towards generic use, and in the Netherlands, we want to add a number of requirements for the Dutch situation with the goal to agree on how to use the CloudEvents specification. In the CloudEvents specification, context attributes are standardized metadata fields that provide essential information about the event itself. The core context attributes defined in CloudEvents include id, source, time, specversion, subject, datacontenttype and so on. These context attributes are typically bundled together as a structured set of metadata fields at the beginning of the event. They form a header-like structure that provides essential information about the event itself. However, they might not always be physically located at the very beginning of the event payload, depending on the transport or serialization format being used.

Discussion Input

This pattern requires additional attention, particularly as we pursue a more event-driven strategy in both the government and energy sectors. AsyncAPI is gaining more importance, and it brings integration camps (synchronous and asynchronous communication) together. New integration technologies and protocols will be introduced, ensuring the sustained relevance of this pattern in the future. To increase operability, allowing more API clients to connect, it can be beneficial to offer APIs in protocols and architectural styles other than REST/JSON. So a protocol agnostic approach is needed. The Context Representation pattern aims to help by normalizing payloads and metadata that producers and consumers can use to ensure logic does not break when the protocol changes. To be continued 😊!

Version Identifier

Known Uses

Versioning is a hotly debated topic in the API community and the book states correctly that the version strategies differ widely and are debated passionately. There are many questions and discussions floating around the internet about the correct way you should version your APIs. The HTTP header versioning (consumer gets to decide which version to request ) appeals to purist RESTafarians. In general, the most common approach is a combination of URI parameters and header criteria. However, in many cases, people tend to fixate blindly on the versioning issue. But the fact that there are so many perspectives on the topic of versioning probably indicates that there isn’t a single universal solution for this issue – there is no single ‘right way’. Perhaps the issue does not solely reside within the versioning strategy, but rather within the change strategy itself. The underlying problem is managing the impact of a change, and the appropriate version strategy can be applied based on this management. See Discussion Input below.

The book explains very well the various methods of implementing versioning to denote progress and maturity in API evolution. In addition to incorporating a Version Identifier in the URI, header, or payload, the more unconventional query-based versioning method is also utilized (however, we have never seen this in the wild).

Some known uses (to mention only a few):

Discussion Input

An interesting observation is that the zealous discussion about the version numbering of an API is a red herring: the version number is not the problem. Instead of a versioning strategy, you should have a change strategy. This observation basically propagates not doing versioning at all, or using it as a last resort since it causes long-term maintenance issues. This can be achieved by making use of hypermedia controls, by reconsidering new resources instead breaking existing ones by applying the correct bounded context boundaries and by developing a smart change strategy that embraces a compatible versioning strategy being always backwards compatible without needless duplication.

When discussing change strategy, we struggle (within Enexis) with balancing API complexity and client-side impact; to what extent should APIs be future-proof (forward-compatible)? As API designers, we often suggest common generic data structures even if a particular consumer does not request this. For instance, even if one consumer is only interested in the numerical value of a total amount we would suggest exposing this in the API as a value and a currency. Hence future-proofing the API for consumers that might require the currency. This example might introduce little extra complexity, but in practice, it’s hard to define guidelines on where future-proofing is helpful and when you are better off accepting a new API version in the future.

The point is that version numbering on a Web API “seems like a simple solution to a simple problem, while in fact it is a very naïve solution to a very complex problem” (quoting Asbjørn Ulsberg). This observation is supported by the following principles: Bertrand Meyer’s open-closed principle suggests that software entities should be open for extension but closed for modification. When applied to APIs, the implication is that you can augment your resources but not change them. You could do more to share the burden of change between API and client. Postel’s law, often referred to as the Robustness Principle, states that you should be liberal in what you accept and conservative in what you send. In terms of APIs, this implies a certain tolerance for consuming services.

More information on this topic can be found here:

Two in Production

Known Uses

The Two in Production pattern is actively employed in the API lifecycle management strategy of the Dutch government. It is interesting to mention that in the DSO API strategy, initially, the support was limited to a maximum of three versions of an API (“three in production”), comprising two major versions and one minor one. Both old and new versions (up to a maximum of three) of an API were concurrently offered for a limited transition period (up to one year), with two major versions available. Through the optional request header, one minor/patch version could be specified. This means that, for instance, in a (pre)production or integration environment, alongside, for example, v1 and v2, there was still the option to access one “older” or, alternatively, one “newer” minor/patch version of these APIs. This version could then be accessed through the request header. This approach was later discontinued due to significant operational complexity: running multiple minor/patch versions in parallel proved unnecessarily complicated. Furthermore, the use of an api-version request header conflicted with employing the version number in the URI. However, it was decided to include the api-version (full version number) in the response headers for every API call, since the URI only contains the major version. This information could then be used for logging, debugging or auditing purposes. In cases where an intermediate networking component returns an error response (e.g., a reverse proxy enforcing access policies), the version number may be omitted.

In the Dutch energy sector, we have opted for a maximum of three versions in production, allowing clients more time to upgrade and adapt within the managed evolution timeframe. This choice aims to strike a good balance between provider complexity and the pace of client adaptation.

Discussion Input

An important question is how many API versions in production should be offered in parallel. As stated very well in the book (page 391): the variant N in production gives clients more time and options to upgrade, but puts more maintenance effort and operational cost on the provider side. As always, it depends, and it is a trade-off between provider maintenance activities and pace of adapting to new major versions by clients. In any case, there should be a clear policy that defines the maximum number of major versions to be offered. The overall objective would be to keep the number of parallel major versions limited.

Public API

Known Uses

Public APIs are open and accessible to anyone who wants to use them. They are often free or have low fees. The goal of Public APIs is to make consumption easy and, very often, to attract as many consumers as possible. Obviously, there are many Public APIs offered in the Dutch Dutch Government Public sector, and they are available and discoverable via public developer portals like:

In these cases of open government data scenarios, very often the Retrieval Operation pattern is applied. In the Dutch Energy sector, there are fewer Public APIs. Some examples:

  • https://www.eancodeboek.nl/ With the help of this service, one can look up a ‘connection identification code’ (EAN code) for a connection to the electricity and gas networks in the Netherlands.
  • https://energieonderbrekingen.nl/api/v2/ Real-time information about gas and power outages. Collaboration of three Dutch grid operators: Enexis Netbeheer B.V., Liander N.V., and Stedin.

There are various ways to categorize and classify APIs. Commonly used is the classification into public, partner, and private. The Dutch government uses the classification internal/external and the further subdivision external/open and external/closed. The latter corresponds to the partner or community classification. See https://geonovum.github.io/KP-APIs/API-strategie-algemeen/Architectuur/#typologie-van-api-s.

It is very well stated in the book (page 49): ‘…the difference between a Public API in general and its Open API variant: a truly open API is a Public API without an API Key or other authentication means…’. See also: https://nordicapis.com/6-types-of-apis-open-public-partner-private-composite-unified/

Discussion Input

Figure 4.8 on page 142 illustrates a Public API in context. It shows a public API with three different consumers: 1) Web App 2) Mobile App 3) Third-Party systems. Within Enexis, we experience different API requirements from different consumers. For instance, frontend-oriented consumers might require form-like API data models that resemble web-forms, whereas Backend systems might require more detailed or generic information. These different requirements can be incorporated into a single Public API. However, the potential risk is creating cluttered APIs with redundant operations and data. How should we manage the requirements of different consumers? When is it justified to fulfill the requirements of different consumers in multiple Public APIs?

Retrieval Operation

Known Uses

Retrieval Operation is a very common pattern and is described in detail and elaborated upon in the book. Next to GraphQL, OData is a query-based API protocol standardized and managed by OASIS: https://www.odata.org/documentation/. Many companies, like SAP and Microsoft, offer OData APIs out of the box. OData provides advanced filer options, and it’s possible to define custom actions to execute robust queries.

Known uses in Dutch government:

  1. Valid: This is a moment in time on which the data returned is valid in reality.
  2. Available: This is a moment in time on which the data returned is available using the same API.
  3. Effective: This is a moment in time on which the data returned became legally effective.

The above time traveling example matches the Time-Bound report variant as described in the book.

Discussion Input

It is difficult to balance client-side flexibility and data model clarity. Some OData services expose large numbers of attributes, making it hard to understand the data model. Creating multiple Retrieval Operations can help scope the data model, but it remains challenging to determine optimal scopes for Retrieval Operations

Another point of discussion is the HTTP verb that should be used for Retrieval Operations It is possible to include a payload in a GET request, but it is not common practice nor recommended according to HTTP specifications. GET requests are primarily used to request data from a specified resource and should not have a payload in the usual sense. For complex queries, the POST operation as catch all method is used instead. But this is using a non-idempotent operation in an idempotent manner. That’s why recently, the QUERY method came into place as a new HTTP method for safe, idempotent query functionality with a request payload.

More information is available online:

Service Level Agreement (SLA)

Known Uses

Service Level Agreements (SLAs) are frequently incorporated into the terms and conditions of services, as denoted in the API Description itself (under “termsOfService” and contact details) within Dutch Government APIs. As legally binding documents, terms and conditions, along with SLAs for public APIs, are drafted, or at least reviewed and approved, by legal experts specializing in the field. However, they often manifest as unstructured and freeform texts, as outlined in the book on page 417.

In Dutch Energy sector the SLA is described in the Service Description Documentation (API user manual that provides service context information, usage examples, details on error handling, etc.) and the following service-level objectives are defined:

  • Response times
  • Availability
  • Uptime and peak load
  • Security
  • Confidentiality
  • Helpdesk availability (service window)

There are no examples where failing to meet an SLA could result in negative financial consequences or punitive penalties (as far as we know).

Examples:

Two examples in our domains are:

As correctly stated on api-patterns.org: “…many public cloud providers have explicit SLAs…”, the following can be added:

Discussion Input

In particular, when considering risk and accountability, notable differences exist between SLAs in the public and private sectors. Public sector SLAs typically encompass risk management associated with public interest, political implications, and adherence to public policy, necessitating elevated levels of accountability and transparency. Conversely, within the private sector, SLAs primarily center on managing financial risks, navigating market competition, and meeting the expectations of shareholders or investors.

ISO/IEC 19086-1 sets a framework to enhance the quality, reliability, and transparency of cloud Service Level Agreements aiming to improve the confidence and trust between cloud service providers and their customers: https://www.iso.org/standard/67545.html.

Wrap Up

Overall, I (Ton) found all patterns in the “API Design Pattern of the Week” series useful and have applied many of them (see above). The book “Patterns for API Design” is a must read for API designers, architects and developers. More comments in this LinkedIn article.

Thank you very much indeed, Ton! Chapter 10 of our book contains two more pattern adoption stories (Table of Content).

Back to homepage