Tech Dive - APIs
Hi Everyone!
Today will be a tech dive on APIs.
We’ll be talking about
What is an API
Internal vs. External APIs
Request-Response APIs
REST
GraphQL
Event-Driven APIs
WebSockets
WebHooks
Feel free to reply back to this email with any questions!
Tech Dive - APIs
What is an API?
An Application Programming Interface (API) is an interface that allows computer applications to exchange data with other computer applications.
These computer applications can be a web server (a web API), an operating system (POSIX) or a software library (tensorflow API for machine learning). These are just a couple examples of computer applications that can be exposed through an API.
When developers talk about APIs, they’re mostly referring to Web APIs (APIs that you can use to talk to a web server).
The rest of this tech dive will be centered on Web APIs.
Internal vs. External APIs
When a company builds an API to expose their application, that API can either be an internal or an external APIs. The internal/external subset is a useful way to differentiate APIs.
Internal API
An internal API is an API that is meant for the company’s own developers. It’s not meant for developers from outside the company to use.
As many companies shift to a Microservices architecture, where your application is divided into a collection of smaller services, the company will have lots of internal APIs that communicate to form the entire application.
If it’s an e-commerce business, you might have a web server that manages the inventory service, a web server for the shipping service and a web server for the storefront application. All these web servers talk to each other through their own APIs.
External API
An external API is an API that a company exposes for other developers to use.
An example is the Spotify API, that you can use for getting information about songs, music playlists, etc.
However, there are increasingly more and more companies that build developer tools, where the API is the product.
An example for this is Stripe, a company that helps developers integrate payment functionality into their apps (so developers can charge users).
These companies will typically charge you based on the number of times you make a call to the API. In Stripe’s case, they charge a payment processing fee (which is typically 2% of the payment + 35 cents).
API Paradigms
Once you build an API, it’s difficult to change it. That’s because changing the API will usually break every application that depends on the API, so it’s a great way to piss off all the API’s users.
This is why it’s extremely important to focus on designing your API the right way the first time. There are several paradigms you can use for designing your API.
Request-Response
Request-Response APIs work with a client requesting information from a server through an API endpoint ( a URL). The server will then send back a response with the information the client requested (or with an error). The data is typically sent as JSON.
The most common paradigm for Request-Response APIs is REST (or a RESTful API) but GraphQL has also become very popular recently.
REST
Representational State Transfer is the most popular choice for building a Web API. It was designed by Roy Fielding in his PhD dissertation in 2000.
Let’s say we have a web server that contains information on movies, like the rating (rotten tomatoes), length, cast, etc.
The website is QuastorMovieDatabase.com and the api is at api.QuastorMovieDatabase.com.
Here’s an example of how you could interact with it’s REST API.
RESTful API
HOST api.QuastorMovieDatabase.com
HTTP GET /movies?name=scarface
The response might be
Using this JSON response, we can filter out exactly what we want client-side.
REST APIs have 6 guiding constraints for how the interface must be designed
Client-Server - REST applications should have a client-server architecture. The client should not be concerned with things like data storage, authentication and business-logic tasks. The server should not be concerned with UI.
Uniform Interface - Uniform Interface defines the interactions between the client and the server. It consists of 4 guiding principles.
Identification of Resources - You use the URL standard to identify a resource. So, with our movies web server, we can identify a particular movie with the URL
/movies?name={movie_name}.Manipulation of Resources - You can use the HTTP standard to manipulate these resources. So,
GETallows you to obtain the data contained by the URL resource. You need the identification (URL) and manipulation (HTTP method).Self-descriptive Messages - The client’s request and the server’s response should both contain all the information necessary for the message to be understood. There should not be additional information in a separate document or in another message.
Hypermedia As The Engine of Application States (HATEOAS) - This relates to how you figure out how to use the API. You should be able to figure it out based off hyperlinks in the responses. An analogy is Wikipedia. When you’re browsing through wikipedia, you can easily navigate to related resources based off the hyperlink structure on the web page. Your REST API should function in the same way. Here’s a more detailed explanation of HATEOAS.
Stateless - All necessary data for the server to handle the request should be contained within the request itself. The server shouldn’t have to look at previous requests from the client when determining how to respond to the current request.
Cacheable - Every response from the server should include whether the response is cacheable and for long responses can be cached by the client. This improves scalability as it reduces the number of requests the client will make to the server (the tradeoff is that it increases the probability of stale data client-side).
Layered System - Between the client and the server, there may be a number of servers that handle different tasks. Load balancing, caching, reverse proxies, etc. These layers should not affect the request or response and the client shouldn’t care about how many layers are between it and the server.
Code On Demand (optional) - This is an optional constraint, but it means that the server can increase the functionality of the client at runtime by sending it code to execute (like JavaScript code).
The vast majority of Web APIs are RESTful. However, another paradigm is gaining steam.
GraphQL
GraphQL is the cool new kid on the block in terms of API Design. It was developed internally by Facebook and released in 2015. Since then, it’s been adopted by companies like GitHub, Yelp and Pinterest.
GraphQL is a query language for APIs that lets you define exactly what pieces of data you want the server to send back. There’s only one API endpoint and your request will define the structure of the data you want from the server.
So, going back to our movies web server. Here’s how it might go.
The response from the server might be something like
The benefits of GraphQL include
Save multiple round trips - With GraphQL, clients can fetch data across resources in a single request. With a traditional REST API, you might have to make multiple HTTP calls to the server, since the resources you need might be at different endpoints. This means preventing under-fetching.
Smaller payload size - With GraphQL, the server will only respond with data that the client specifically asks for. With a REST API, servers will often respond with data that the client didn’t need, just because that’s how the API was designed. This means preventing over-fetching.
Avoid Versioning - If you noticed with our GraphQL example, it isn’t really dependent on any version. There’s a single API endpoint and we just request the resources that we need. As long as that one API endpoint doesn’t change (and the resource names don’t change), the provider can make whatever changes he wants.
The cons of GraphQL are
Performance Issues for Complex Queries - Remember that GraphQL is just a query language. This query is then translated into a query for the database, and then the data is sent back. If the API user sends complex GraphQL queries (with lots of nesting for example), that may result in a very complex (and slow!) database query. This can lead to performance issues.
Caching Complexity - A REST API provides many different endpoints which leads to a pretty easy caching system where you match URL endpoints. However, GraphQL provides a single endpoint with many different queries. This makes caching much more complicated compared to REST. You can read about caching in Apollo (implementation of GraphQL) here.
There’s an amazing documentary on the creation of GraphQL that you can check out here. I’d highly recommend watching it if you have a spare 20 minutes.
Event Driven APIs
Let’s say you’re a big fan of Billie Eilish. You’re such a massive fan, that you want to be alerted whenever she publishes a new music video on YouTube!
Then, you can go and comment “first!” on the video.
So, you decide to write a script that will send you a text message whenever she publishes a new song on YouTube.
Your script will need to communicate with YouTube’s API to get data on her latest video. But how should you implement this?
Under the Request-Response paradigm, your script will have to constantly poll YouTube server’s with a HTTP GET request, where it gets Billie Eilish’s latest video.
If the video was just uploaded, then your script will send you a text.
Remember, you have to comment “first!” within the first 30 seconds of the video being published if you want to have a shot at the first comment. Therefore, your script will have to be sending multiple requests every minute to Google’s servers.
Also, Billie uploads a new video every 2-3 weeks, so the vast majority of your requests will come back as failures.
Doing something like this is called Polling the API. WIth polling, a developer is constantly querying an API endpoint at a predetermined frequency to look for new data.
This is a huge waste of resources. What would be better is if Google’s server could alert your script when the event happens (Billie Eilish uploads a video) instead of you constantly polling Google’s servers.
This is where Event Driven APIs come in.
Two ways of building an Event Driven API are with WebHooks and WebSockets.
By the way, YouTube’s API uses WebHooks to implement the functionality that we’re talking about. You can check it out here.
WebHooks
You can think of a WebHook as a “reverse” of Request-Response. With Request-Response, the server will have an API endpoint and the client will send an HTTP request.
With WebHooks, the client will have an API endpoint, and the server will send an HTTP request to it. The API provider implementing the WebHook will send a POST message to the configured URL when the event happens.
Here’s a great illustration
You can configure a WebHook by first setting up your own web server so you can recieve messages.
Then, give the API provider an endpoint and configure what kind of messages you want them to send you. Here’s an example with GitHub’s Webhook.
Now, there are some important considerations with WebHooks…
Security - As the developer of the endpoint, it is your responsibility to make sure you’ve received a legitimate WebHook. Any random person can send HTTP requests to your endpoint, so you have to verify that the requests are actually coming from the API provider. One way of doing this is with verification tokens.
Failures - You server will have occasional failures. Therefore, most WebHook providers will retry delivery in the event of a failure. Your system should be setup to handle this.
But, what if you don’t want to open up an HTTP endpoint? This is where WebSockets come in…
WebSockets
The WebSocket specification creates an API for the web that mimics “socket” functionality. In other words, it creates a persistent connection between the client and server so both parties can send data over at any time (the connection is full-duplex, so both parties send messages simultaneously if necessary).
Creating a WebSocket connection is quite simple. You’ll just run something like the following on your client…
var exampleSocket = new WebSocket(url);
To send data to the server ( or to the client if you’re the server), you just run
exampleSocket.send(“Wazup”)
Receiving messages can be done with the event handler onMessage
exampleSocket.onMessage = function(event){
console.log(“handle ” + event)
}
There are various other event handlers like onOpen so you can run a function once your connection opens or onError so you can handle errors.
WebSockets work great for fast, live streaming data and long-lived connections.
Issues with WebSockets
Need a Persistent Connection - With WebSockets both the client and the server need to keep the connection alive. So if your clients have a poor internet connection, they may keep dropping the connection and must go and reinstate it. This may add unnecessary load to your server.
Scalability - If 1,000,000 people want to use your app simultaneously, that means that you’ll have to manage 1,000,000 connections on your server (because WebSocket connections are persistent). This can get hairy, especially when the connections aren’t idle ( clients are sending a lot of messages to your server or vice versa). Additionally, horizontal scaling with WebSockets is a lot more tricky than it is for a REST API or WebHooks because you have to keep track of which server opened which connection. You can read more about the difficulties here.
Hope you enjoyed this tech dive!
Feel free to reply back with any feedback/comments.