How WhatsApp scaled to 1 billion users with only 50 engineers

Today we’ll be talking about

• How WhatsApp scaled to 1 billion users with only 50 engineers.

  • WhatsApp’s engineering culture of small, simple and staying mission-oriented.

  • The company’s tech stack (Erlang, FreeBSD, SoftLayer)

• How to quickly learn new programming languages and frameworks

  • This is from a recent discussion on Hacker News. We summarized some of the best comments.

• Plus, some tech snippets on

  • An in-depth dive to Hadoop Distributed File System (HDFS)

  • Concurrency in Java vs. Go

  • An amazing free textbook on math for programmers (covers Calculus, Linear Algebra and Discrete Math)

Plus, we have a solution to our last coding interview question on creating a Spiral Matrix and a new interview question from Facebook.

---

Quastor Daily is a free Software Engineering newsletter sends out Technical Deep Dives, summaries of Engineering Blog Posts, and FAANG Interview questions (with detailed solutions).

---

How WhatsApp served 1 billion users with only 50 engineers.

In 2016, WhatsApp reached more than a billion users and had the following load stats

• 42 billion messages sent daily

• 1.6 billion pictures sent daily

• 250 million videos sent daily

They managed to serve this scale with only 50 engineers.

Here’s a dive into the engineering culture and tech stack that made this possible.

Engineering Culture

WhatsApp’s Engineering culture consists of 3 main principles

1. Keep Things Small

2. Keep Things Simple

3. Have a Single Minded Focus on the Mission

Keep Things Small

WhatsApp consciously keeps the engineering staff small to only about 50 engineers.

Individual engineering teams are also small, consisting of 1 - 3 engineers and teams are each given a great deal of autonomy.

In terms of servers, WhatsApp prefers to use a smaller number of servers and vertically scale each server to the highest extent possible.

Their goal was previously to have 1 million users for every server (but that’s become more difficult as they’ve added more features to the app and as users are generating more activity on a per-user basis).

Having a fewer number of servers means fewer things breaking down, which makes it easier for the team to handle.

The same goes for the software side where they limit the total number of systems and components in production.

That means fewer systems that have to be developed, deployed and supported.

There aren’t many systems/components that are developed and then put into maintenance mode (to eventually become orphans until something goes wrong).

Keep Things Simple

WhatsApp uses the mantra Just Enough Engineering.

They avoid over-investing in systems and components.

Instead, they focus on building just enough for scalability, security and reliability.

One of the key factors when they make technical choices is “what is the simplest approach?”

Also, they avoid investing in automation unless it’s completely necessary.

Have a Single Minded Focus on the Mission

Product Design at WhatsApp is incredibly focused.

It’s dedicated to delivering a core communications app with a great UI.

They avoid extra bells and whistles and don’t implement features that aren’t exclusively focused on core communications.

The simpler product makes it much easier to maintain and scale.

Tech Stack

The tech stack revolves around 3 core components: Erlang, FreeBSD and SoftLayer.

Erlang

Erlang is the programming language of choice for WhatsApp’s backend systems.

Erlang was designed for concurrency from the start, and fault tolerance is a first class feature of the language.

You can read more about Erlang’s fault tolerance here.

Developer productivity with Erlang is also extremely high. However, it is a functional language, so it takes a little getting used to if you’re not familiar with the paradigm.

The language is very concise and it’s easy to do things with very few lines of code.

The OTP (Open Telecom Platform) is a collection of open source middleware, libraries and tools for Erlang.

WhatsApp tries to avoid dependencies as much as possible, but they do make use of Mnesia, a distributed database that’s part of OTP.

Erlang also brings the ability to hotswap code. You can take new application code and load it into a running application without restarting the application.

This makes the iteration cycle very quick and allows WhatsApp to release quick fixes and have extremely long uptimes for their services.

To see exactly how WhatsApp’s backend is built with Erlang, you can watch this talk from 2018.

FreeBSD

FreeBSD is the OS WhatsApp uses for their servers.

The decision to use FreeBSD was made by the founders of WhatsApp, based on their previous experience at Yahoo!

The founders (and a lot of the early team) all used to be part of Yahoo!, where FreeBSD was used extensively.

To see exactly how WhatsApp uses FreeBSD, you can watch this talk.

Just note, the talk is from 2014, so some things may be out of date now.

SoftLayer

SoftLayer is the hosting platform that WhatsApp was using in 2016.

They picked SoftLayer for two main reasons

1. The availability of FreeBSD as a first class operating system.

2. The ability to order and operate bare metal servers.

However, SoftLayer is owned by IBM (part of IBM public cloud), and WhatsApp has since moved off SoftLayer to use Facebook’s infrastructure.

They made the transition in 2017.

You can view the full talk on WhatsApp’s engineering here.

Get more specific details from this High Scalability post on WhatsApp Engineering.

---

Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.

---

How to learn new programming languages and frameworks quickly

This is an interesting discussion on Hacker News for a process to learn new things quickly.

Here are some of the top answers summarized.

• For a new programming language, there are a couple of standard things you should try to implement.

  Write several programs where you

  • Read/write to a file

  • Turn a structured object into JSON

  • Parse JSON into an object

  • A basic script that can be run from the CLI, parses flags/args, reads stdin

  • Send a HTTP request

  • Implement a very basic web server

  When you’re doing these exercises, do not copy/paste any code.

  You should be typing out the code yourself.

• Try to bias towards reading (blogs, documentation, code, books, etc.) over watching video tutorials. Reading is a lot faster than watching video and you can be far more efficient with your time.

• A 3 step process for learning new frameworks is to divide it up into a week’s work (10 - 15 hour total commitment)

  • Prepare - Spend 1 hour Monday through Thursday reading documentation, books, and watching content. This way you’ll have time to sleep on the concepts.

  • Plan - On Friday, spend 1 hour preparing a small project idea for the weekend that will use all the concepts you learned over the week.

  • Project - Spend 4 - 6 hours building the project over the weekend.

---

Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.

---

Tech Snippets

• A Programmer's Introduction to Mathematics - If you’re looking to learn more about areas like machine learning, you might have to brush up on your math background. Jeremy Kun wrote a fantastic textbook on math for programmers that covers topics in Calculus (Single and Multivariable), Linear Algebra, and Discrete Math.

  You can set your own price when you purchase the e-book, so you can pay anything from $0 to $100.

• Why you can have millions of Goroutines but only thousands of Java Threads

  If you’ve been working with JVM based languages for a while, you’ve probably come across a situation where you’ve reached the limit of the number of concurrent threads you can have.

  On your personal computer, this limit is usually around ~10,000 threads.

  On the other hand, you can have more than a hundred million goroutines on a laptop with Go.

  This article explores why you can have so many more Goroutines than threads.

  There’s two main reasons why

  1. The JVM delegates threading to operating system threads. Go implements its own scheduler that allows many Goroutines to run on the same OS thread.

  2. The JVM defaults to a 1 MB stack per thread. Go’s stacks are dynamically sized and a new goroutine will have a stack of about 4 KB.

• An in-depth dive into HDFS (Hadoop Distributed File System)

  The article describes the architecture of HDFS and also goes into the experience of using HDFS to manage 40 petabytes of data at Yahoo!

---

Interview Question

Implement a BST Iterator class that represents an iterator over the in-order traversal of a Binary Search Tree.

Implement the following methods

• BSTIterator - constructor. The root of the BST will be passed in as a parameter. The pointer should be initialized to a non-existent number small than any number in the BST.

• hasNext - Returns true if there exists a number in the traversal to the right of the pointer, otherwise returns false.

• next - Moves the pointer to the right, then returns the number at the pointer.

---

Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.

---

Previous Solution

As a reminder, here’s our last question

Given a positive integer i, generate an nxn matrix filled with elements from 1 to i^2 in spiral order.

Example

Input: i = 3

Output: [[1,2,3],[8,9,4],[7,6,5]]

Here’s the question in LeetCode.

Solution

We can solve this question by creating an i x i matrix and then looping through it in a spiral fashion and setting each element to a value from 1 to i^2.

In order to loop through in spiral fashion, we’ll first create 4 variables: left, right, top, bot.

These 4 variables represent pointers to the 4 different sides of our matrix (left column, right column, top row, bottom row).

Left will point to the 0th column, right will point to the last column, top will point to the 0th row and bottom will point to the last row.

When we iterate through our matrix in spiral order, we’ll do it in the following way

1. Iterate through the top row. Now, increment the top row.

2. Iterate through the right column. Now, decrement the right column.

3. Iterate through the bottom row. Now, decrement the bottom row.

4. Iterate through the left column, Now, increment the left column.

We’ll iterate through these steps while the left pointer is less than or equal to the right pointer and the top pointer is less than or equal to the bottom pointer.

When our loop terminates, we can return the completed matrix.

Here’s the Python 3 code.

---

Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.