Observing Phoenix on Heroku
In this post, I'm going to show you how recent updates to the Heroku command line interface allows one to use Erlang's Observer on Phoenix apps running on Heroku.
I wish I had time to write a lengthy blog post on using Observer, why it wasn't possible on Heroku before, and why it is now, but I don't. Instead, I'm just going to give a brief overview of what's now possible due to recent changes in the CLI.
In order to connect directly to an Erlang node running your Phoenix
application, two things typically are needed. First, your node must be
registered with the Erlang Port Mapper
Daemon (or epmd
) that you connect to
locally. Second, your node must be listening on a port that you can connect
to directly.
The Heroku command line interface has a command called heroku ps:forward
that forwards a local port to a port on your dyno. Until recently, though,
this command could only forward a single port at a time. What this meant
was that you couldn't simultaneously forward your local epmd
client to
epmd
on the dyno and also forward a local port to the port your node
listens on for distributed connections.
Recently, however, with the help of Herokai Joe
Kutner, I was able to land a pull
request in the Heroku
command line interface that updates heroku ps:forward
to accept a
comma-separated list of ports to forward.
In order for this to work, a few changes to a normal Phoenix deploy to Heroku are necessary. You can see these changes in my jclem/phoenix-template repository, or you can generate your own project from the template and deploy it directly to Heroku.
First, the application's Procfile needs to set a few options that allows us to easily make remote connections:
web: elixir --cookie $OTP_COOKIE --name server@127.0.0.1 --erl '-kernel inet_dist_listen_min 9000' --erl '-kernel inet_dist_listen_max 9000' -S mix phx.server
This is a lengthy Procfile command, so here's the command itself in a more readable form:
$ elixir --cookie $OTP_COOKIE \
--name server@127.0.0.1 \
--erl '-kernel inet_dist_listen_min 9000' \
--erl '-kernel inet_dist_listen_max 9000' \
-S mix phx.server
This new Procfile sets the distributed Erlang cookie to the value of an OTP_COOKIE
environment variable. It also gives your node a predictable name to connect to (server@127.0.0.1
) and ensures that the node will listen for remote connections on port 9000
(we set the port min and max to 9000
, so it will only be that value).
Once your app is running, you can easily connect directly to your dyno. First, in one shell, forward a couple of ports directly to your dyno:
$ heroku ps:forward 9001:4369,9000
This forwards your local port 9001
to the default port that epmd
will be listening on on the dyno, which is 4369
. Next, it forwards your local port 9000
to port 9000
on the dyno, which is where the node is listening for remote connections.
In another shell, you can now start a console, connect to the remote node, and open observer (you'll need to get the value of the OTP_COOKIE
secret on the dyno and use it here):
$ env ERL_EPMD_PORT=9001 iex --cookie $OTP_COOKIE --name console@127.0.0.1
Erlang/OTP 22 [erts-10.4.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Interactive Elixir (1.9.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(console@127.0.0.1)1> Node.connect(:"server@127.0.0.1")
true
iex(console@127.0.0.1)2> :observer.start
:ok
Once the Observer starts, you can use the "Nodes" dropdown menu to select
your remote node (server@127.0.0.1
) and begin observing!