Understanding Elixir OTP Applications - Part 5, Building High Availability Systems

In this series we've taken a look at how Elixir communicates through messages to deliver distributed systems, how supervisor trees watch over processes to deliver fault tolerant systems, how the process architecture delivers soft real time systems, and lastly, how the BEAM VM manages hot swapping.

The team that created Erlang understood that it's not realistic to build perfect applications. We all make mistakes. The genius of Erlang is that it doesn't matter. The language saves you from yourself. If you create a buggy module it won't block any other processes from performing their duties. Erlang also comes packaged with a lot of really useful debugging tools making it possible to find out where the problems are coming from. This is really important for Erlang as bad modules don't block other processes so from the outside it can seem like nothing is wrong. 

Erlang ships with a utility called the Observer. We can run this observer from iEX to get an overview of our system. To open it you simply run:

:observer.start

Running this command will open a GUI with a lot of information about processes currently alive for the current VM.

From the Load Charts tab you can view information relating to memory usage, IO, and scheduler utilization over time.

From the Applications tab you can see all of the processes alive under Kernel. Clicking on one of these nodes will reveal more information specific to that process. From here you can see useful information like Reductions which shows an approximation of how long the process has been alive. If you're running into unexpectedly high CPU usage,  this is a good place to hunt.

We can also run the observer against a remote node by opening a remote iEX shell between your local machine and the node you want to observe. To do so you need the IP for the machine and the cookie which is commonly found in ~/.erlang.cookie

iex --name "joe@joes_mbp" --cookie my_apps_cookie --remsh "my_app@172.217. 22.14"

From here you can start the observer and view the information related to the node.

With the tools we've already discussed we can build a system that will tolerate faults, run massive amounts of processes very efficiently, and give us tools to hunt and fix bugs. In software, there typically comes a day when you have to scale. These mechanisms make it very easy to scale your Erlang and Elixir applications. Erlang is so well-equipped to scale, WhatsApp famously scaled to 900 million users with a 50 person team. 

BEAM is very well equipped to scale vertically, i.e. add more resources to a single machine due to the nature of the schedulers. Each CPU core is assigned a scheduler so if you are running into limits of how many processes are being fulfilled in a reasonable timeframe, you can simply add more cores. You might also run into memory issues, in which case you can simply increase the RAM on the machine.

So if we hit the limits for vertical scaling - can we scale horizontally? Yes. In fact, the same messaging systems that are used to communicate with processes on the current machine are also designed to talk with other machines within the same cluster. Libraries such as libcluster can be used to form nodes that auto-discover each other. 

Erlang's approach to scalability is not an after-thought. It's built in from the ground up. Using all of the features discussed in this series we can build high availability applications with confidence. Erlang and Elixir lend themselves very well to web software with their super high concurrency rates.
In 2018, Cisco announced that 90% of all internet traffic passes through Erlang controlled nodes. In 2015 the Phoenix Framework proved they could serve 2 million concurrent web-sockets from a single 40-core, 128Gb machine. There are multiple mind-blowing success stories and systems that are wildly efficient running Erlang or Elixir. There are also a lot of very talented developers contributing within the community.

Personally, the thing that makes me most excited with Erlang is the stuff that the language solves right out of the box. As developers, we spend a great deal of our time ensuring that our code works around issues. With Erlang we don't need to do that so much. I'm super excited about Elixir's future. I love working with Phoenix and also love functional programming. After spending years with ruby I'm happy to say I've found a new language that I absolutely love.