Home

Phoenix is Not Magic

Note: This series is using Phoenix 1.4.1

A common criticism leveled against Phoenix is that it’s just another monolithic framework like Rails. Or that it’s too magic, or too opinionated. Generally, these complaints are unfounded, and seem to stem from superficial similarities (Elixir does look a little Ruby-ish if you squint) rather than experience. This blog post is intended to be the first of several posts that can serve as a response — a light overview of core Phoenix internals to demystify your application.

The rough outline of these posts will look something like this:

Part 1 - Endpoint
Part 2 - Router through Controller
Part 3 - Rendering views
Part 4+ - Socket, Repo, etc…

This first part will cover the “Endpoint”. If you are new to Phoenix, you may only recognize the Endpoint as “the thing you sometimes tweak configs in”, but it is essential to the function of your application.

High Level Overview

If we generate a new Phoenix project with mix phx.new phoenix_internals, we’ll find a module called PhoenixInternalsWeb.Endpoint that looks something like this:

# lib/phoenix_internals_web/endpoint.ex

defmodule PhoenixInternalsWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :phoenix_internals

  socket "/socket", PhoenixInternalsWeb.UserSocket,
    websocket: true,
    longpoll: false

  # Serve at "/" the static files from "priv/static" directory.
  #
  # You should set gzip to true if you are running phx.digest
  # when deploying your static files in production.
  plug Plug.Static,
    at: "/",
    from: :phoenix_internals,
    gzip: false,
    only: ~w(css fonts images js favicon.ico robots.txt)

  # Code reloading can be explicitly enabled under the
  # :code_reloader configuration of your endpoint.
  if code_reloading? do
    socket "/phoenix/live_reload/socket", Phoenix.LiveReloader.Socket
    plug Phoenix.LiveReloader
    plug Phoenix.CodeReloader
  end

  plug Plug.RequestId
  plug Plug.Logger

  plug Plug.Parsers,
    parsers: [:urlencoded, :multipart, :json],
    pass: ["*/*"],
    json_decoder: Phoenix.json_library()

  plug Plug.MethodOverride
  plug Plug.Head

  # The session will be stored in the cookie and signed,
  # this means its contents can be read but not tampered with.
  # Set :encryption_salt if you would also like to encrypt it.
  plug Plug.Session,
    store: :cookie,
    key: "_phoenix_internals_key",
    signing_salt: "iiFYGc1q"

  plug PhoenixInternalsWeb.Router
end

This module is the heart of our Phoenix application. It’s also little more than a standard Plug¹ configuration. In reality, a lot of what looks like “Phoenix functionality” is really just Plug underneath. In this case, we’re looking at the work of Plug.Builder, which allows us to create plug pipelines.

Plugs in the pipeline are defined with the plug macro, then are called in the order that they are defined. Modules that use Plug.Builder are plugs themselves, and are called accordingly.

Here is a Plug.Builder example modified from the Plug documentation:

defmodule MyApp do
  use Plug.Builder

  plug Plug.Logger
  plug :hello, upper: true

  def hello(conn, opts) do
    body = if opts[:upper], do: "WORLD", else: "world"
    send_resp(conn, 200, body)
  end
end

The structure and intention of this example and our Phoenix Endpoint is quite similar. In fact, we can see that if we just started moving plug definitions from our Endpoint to this pipeline, we’d very quickly wind up with the same functionality. The only piece the Phoenix Endpoint is missing is use Plug.Builder, which is ultimately included in the Endpoint via use Phoenix.Endpoint.

We will get back to this momentarily, but first as an exercise, let’s remove all of the Phoenix-specific macros from the Endpoint - that is the socket, router, and code reloading blocks. Then let’s add the following to the end of our module:

# lib/phoenix_internals_web/endpoint.ex

plug :hello

def hello(conn, _) do
  send_resp(conn, 200, "Hello, world!")
end

If we start up our Phoenix application and go to http://localhost:4000, we’ll see “Hello, world!”. We can even take this a step further with a naive router.

# lib/phoenix_internals_web/endpoint.ex

plug :hello

def hello(conn, _) do
  body =
    case conn.request_path do
      "/" -> "Hello, world!"
      "/bye" -> "Goodbye, world!"
      _ -> "What?"
    end

  send_resp(conn, 200, body)
end

Try going to http://localhost:4000/bye or http://localhost:4000/nothing to see our custom router in action. This is a fully functioning (if very basic) Phoenix web application, bootstrapped off of simple Plug elements.

We can see that the Endpoint is just Plug, and, from our understanding of Plug, we know that somewhere in the codebase the following steps must occur:

Something calls PhoenixInternalsWeb.Endpoint.call(conn, opts)
The conn and opts find their way to Plug.Builder.
Plug.Builder does its thing and calls all of the plugs in the Endpoint in order.

This is already a pretty solid understanding of how the Endpoint works, but we are still missing a few details. How is the application actually started? How do we get from mix phx.server to a running application?

The Internals

If you’ve built a web application with Phoenix, you know your application starts with a call to mix phx.server. This is a very basic use of Mix, Elixir’s application build tool. If a custom Mix task is called “phx.server,” you can expect to find a “phx.server.ex” file somewhere in your project directory². In this file, you’ll find a very simple task definition.

# deps/phoenix/lib/mix/tasks/phx.server.ex

  def run(args) do
    Application.put_env(:phoenix, :serve_endpoints, true, persistent: true)
    Mix.Tasks.Run.run run_args() ++ args
  end

  defp run_args do
    if iex_running?(), do: [], else: ["--no-halt"]
  end

This definition sets up some environment variables, but essentially just runs Mix.Tasks.Run, a built-in task that runs the current application. Let’s take a look at our application configuration in “mix.exs.”

When we make an OTP application with Elixir, we can define our application parameters through the application/0 function in “mix.exs.” If we’ve used the Phoenix generators, that function will look like this:

# mix.exs

  def application do
    [
      mod: {PhoenixInternals.Application, []},
      extra_applications: [:logger, :runtime_tools]
    ]
  end

The part that is most interesting to us is the mod key, which specifies the Application callback module. When the application is started, this is the module that will be invoked. From our application definition, we can expect the call flow to look something like…

You run mix phx.server, which in turn starts the application.
Some internal Mix/Elixir thing calls Application.start(:phoenix_internals).
Our callback configuration results in a call to PhoenixInternals.Application.start(_type, _args).

Let’s a look at PhoenixInternals.Application. As expected, this is your Application callback module, and it defines the necessary start/2 function.

# lib/phoenix_internals/application.ex

  def start(_type, _args) do
    # List all child processes to be supervised
    children = [
      # Start the Ecto repository
      PhoenixInternals.Repo,
      # Start the endpoint when the application starts
      PhoenixInternalsWeb.Endpoint
      # Starts a worker by calling: PhoenixInternals.Worker.start_link(arg)
      # {PhoenixInternals.Worker, arg},
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: PhoenixInternals.Supervisor]
    Supervisor.start_link(children, opts)
  end

The Application spawns a supervisor, PhoenixInternals.Supervisor, which starts and monitors the Ecto repository, PhoenixInternals.Repo, and the Phoenix Endpoint, PhoenixInternalsWeb.Endpoint. This is all standard Elixir/OTP stuff - if you’re lost, the documentation in Elixir tends to be stellar, and the Phoenix source is well commented with links back to the docs.

For the moment, we are going to ignore Ecto, and focus on PhoenixInternalsWeb.Endpoint. Again, if we have some knowledge about the way Elixir and OTP work, we can have some pretty strong assumptions about what is happening in the Endpoint. In particular, we know that, as a child of a Supervisor, the Endpoint module will have to define a child_spec which will dictate the way the process is started.

If we take a look at PhoenixInternalsWeb.Endpoint again, we won’t see the definition we are looking for, but we will see a use Phoenix.Endpoint, otp_app: :phoenix_internals. Now, calling use just runs the used module’s __using__ macro at compile time. In turn, the __using__ macro includes the quoted source in the calling module.

Knowing this, we can expect Phoenix.Endpoint to work (if not look) like the following:

defmodule Phoenix.Endpoint do
  defmacro __using__(opts) do
    quote do
      def child_spec(opts) do
        child_spec_map
      end
    end
  end
end

Let’s take a look at the module now, and see how our expectations match up. A quick search in the Phoenix.Endpoint module will reveal the __using__ macro.

# deps/phoenix/lib/phoenix/endpoint.ex

defmacro __using__(opts) do
  quote do
    @behaviour Phoenix.Endpoint

    unquote(config(opts))
    unquote(pubsub())
    unquote(plug())
    unquote(server())
  end
end

This looks a bit different from what we expected, but presumably the child_spec function is defined in one of these included function calls. Perusing each function in turn, or searching for child_spec, shows that we are correct. child_spec is defined in the server function.

The server function defines a number of other functions that will be included in our own Endpoint, but there are only two that we really care about right now: child_spec and start_link.

# deps/phoenix/lib/phoenix/endpoint.ex

defp server() do
  quote location: :keep, unquote: false do
    
    def child_spec(opts) do
      %{
        id: __MODULE__,
        start: {__MODULE__, :start_link, [opts]},
        type: :supervisor
      }
    end

    def start_link(_opts \\ []) do
      Phoenix.Endpoint.Supervisor.start_link(@otp_app, __MODULE__)
    end
    
    ...snip...
  end

If you aren’t familiar with the quote/unquote macros, I’d suggest taking a look at the documentation, as it is beyond the scope of this post. Although quote/unquote is the core of Elixir’s metaprogramming, you only need a superficial understanding to follow along with the source.

Anyway, back to the code. We were looking for the child_spec function, and we’ve found it. We can see that it’s defining the callback function as {__MODULE__, :start_link, [opts]}. Because of metaprogramming, __MODULE__ will expand to our own Endpoint module, meaning that PhoenixInternalsWeb.Endpoint.start_link/1 is our callback. Luckily for us, start_link is defined just below child_spec.

This function is very simple, and appears to be kicking off an Endpoint supervisor. The __MODULE__ parameter will expand to the current module, and, in this case, @otp_app will be :phoenix_internals. The “how” on the @otp_app definition is something we will cover a little later. For now, let’s take a look at the Endpoint Supervisor.

At the very top of the Supervisor module, you’ll find the start_link definition.

# deps/phoenix/lib/phoenix/endpoint/supervisor.ex

def start_link(otp_app, mod) do
  case Supervisor.start_link(__MODULE__, {otp_app, mod}, name: mod) do
    {:ok, _} = ok ->
      warmup(mod)
      ok
    {:error, _} = error ->
      error
  end
end

This function calls the built-in supervisor start_link, which triggers the init callback, which will return the appropriate specifications. The init function is quite large, but the interesting/important bit is at the bottom of the function definition.

# deps/phoenix/lib/phoenix/endpoint/supervisor.ex

def init({otp_app, mod}) do
  ...snip...

  children =
    config_children(mod, secret_conf, otp_app) ++
    pubsub_children(mod, conf) ++
    socket_children(mod) ++
    server_children(mod, conf, otp_app, server?) ++
    watcher_children(mod, conf, server?)

  # Supervisor.init(children, strategy: :one_for_one)
  {:ok, {{:one_for_one, 3, 5}, children}}
end

A quick scan of this code, and the intention is quite clear. Specs are being created for config, pubsub, socket, server, and watcher child processes, which will be supervised by our supervisor. Since, at the moment, we are trying to figure out how Phoenix is starting and running a web server, server_children is where we’ll look next.

# deps/phoenix/lib/phoenix/endpoint/supervisor.ex

defp server_children(mod, config, otp_app, server?) do
  if server? do
    user_adapter = user_adapter(mod, config)
    autodetected_adapter = cowboy_version_adapter()
    warn_on_different_adapter_version(user_adapter, autodetected_adapter, mod)
    adapter = user_adapter || autodetected_adapter

    for {scheme, port} <- [http: 4000, https: 4040], opts = config[scheme] do
      port = :proplists.get_value(:port, opts, port)

      unless port do
        raise "server can't start because :port in #{scheme} config is nil, " <>
                "please use a valid port number"
      end

      opts = [port: port_to_integer(port), otp_app: otp_app] ++ :proplists.delete(:port, opts)
      adapter.child_spec(scheme, mod, opts)
    end
  else
    []
  end
end

We are now several layers deep into function calls, so this function may start to feel a bit confusing. If it helps, in our generated application, the parameters will resolve to…

server_children(PhoenixInternalsWeb.Endpoint, config, :phoenix_internals, true)

All this function is doing is fetching the server adapter from the configs, and then fetching the child spec from the adapter for HTTP and HTTPS configurations. In a brand new project, unless you specifically define an adapter, it’s going to default to Cowboy2Adapter.

The child_spec function in the Cowboy2Adapter module is one of the roughest pieces of code in the Phoenix codebase. It is a collection of configuration options, and, because it is purely internal, it isn’t well documented. Fortunately, the function is quite small.

# deps/phoenix/lib/phoenix/endpoint/cowboy2_adapter.ex

def child_spec(scheme, endpoint, config) do
  if scheme == :https do
    Application.ensure_all_started(:ssl)
  end

  dispatches = [{:_, Phoenix.Endpoint.Cowboy2Handler, {endpoint, endpoint.init([])}}]
  config = Keyword.put_new(config, :dispatch, [{:_, dispatches}])
  spec = Plug.Cowboy.child_spec(scheme: scheme, plug: {endpoint, []}, options: config)
  update_in spec.start, &{__MODULE__, :start_link, [scheme, endpoint, &1]}
end

The initial child_spec creation is deferred to Plug, with its built-in Cowboy adapter. Of particular interest to us, in the config options, is the dispatch key, which is set to Phoenix.Endpoint.Cowboy2Handler. This is a Cowboy/Ranch configuration, and means that the Cowboy2Handler will be acting as Cowboy handler middleware.

The final statement in the function, overrides the default Plug start key and sets start_link in our current module as the mechanism for starting the child process. In turn, our start_link will call :ranch_listener_sup.start_link() to kick off a Ranch supervisor.

How all of this actually works is Cowboy and Plug internal, and beyond the scope of this post. However, a nice way to get a picture of all of this is by running Observer.

$ iex -S mix phx.server
(iex)> :observer.start()

Using Observer, you can see that our supervision tree looks something like this:

                 ┌────────────────────────────────────────┐   
                 │                                        │   
                 │                                        │   
                 │                                        │   
                 │                                        │   
                 │   Elixir.PhoenixInternals.Supervisor   │   
                 │                                        │   
                 │                                        │   
                 │                                        │   
                 │                                        │   
                 └───┬───────────────────────────────┬────┘   
                     │                               │        
                     │                               │        
                 ┌───┘                               └─┐      
                 │                                     │      
                 │                                     │      
┌────────────────┴──────────────────────┐        ┌─────┴─────┐
│                                       │        │           │
│                                       │        │           │
│  Elixir.PhoenixInternalsWeb.Endpoint  │        │           │
│                                       │        │ Stuff We  │
│                                       │        │  Aren't   │
└────────────────┬──────────────────────┘        │  Worried  │
                 │                               │About Right│
                 │                               │    Now    │
┌────────────────┴────────────────┐              │           │
│                                 │              │           │
│                                 │              │           │
│       ranch_listener_sup        │              └───────────┘
│                                 │                           
│                                 │                           
└────────────────┬────────────────┘                           
                 │                                            
                 │                                            
┌────────────────┴────────────────┐                           
│                                 │                           
│                                 │                           
│       ranch_acceptors_sup       │                           
│                                 │                           
│                                 │                           
└────────────────┬────────────────┘                           
                 │                                            
  ┌─────────┬────┴────┬────────┐                              
  │         │         │        │                              
┌─┴─┐     ┌─┴─┐     ┌─┴─┐    ┌─┴─┐                            
│   │     │   │     │   │    │   │                            
└───┘     └───┘     └───┘    └───┘                            
                                                              
            acceptors

Back to the Cowboy2Handler. What does it mean that it will be acting as Cowboy middleware? Well, basically, it means that every request that passes through Cowboy will result in a call to Cowboy2Handler.init(req, {endpoint, opts}).

# deps/phoenix/lib/phoenix/endpoint/cowboy2_handler.ex

def init(req, {endpoint, opts}) do
  conn = @connection.conn(req)

  try do
    case endpoint.__handler__(conn, opts) do
      {:websocket, conn, handler, opts} ->
        ...snip...

      {:plug, conn, handler, opts} ->
        %{adapter: {@connection, req}} =
          conn
          |> handler.call(opts)
          |> maybe_send(handler)

        {:ok, req, {handler, opts}}
    end
  catch
    ...snip...
  after
    receive do
      @already_sent -> :ok
    after
      0 -> :ok
    end
  end
end

The first thing Phoenix does is turn Cowboy’s request map into a Plug.Conn structure. If you want to see this for yourself, you can add IO.inspect(req) and IO.inspect(conn) just after the conn assignment, and recompile your dependencies with mix deps.compile. When you rerun your application, you will see each request come through.

The next thing this init function does is call endpoint.__handler__. Again, the referenced endpoint variable is our own Endpoint, which uses Phoenix.Endpoint, so we can take a look at “endpoint.ex” to find the __handler__ function. You’ll find __handler__ grouped with several other functions.

# deps/phoenix/lib/phoenix/endpoint.ex

def __handler__(%{path_info: path} = conn, opts), do: do_handler(path, conn, opts)

unquote(instrumentation)

unquote(dispatches)
defp do_handler(_path, conn, opts), do: {:plug, conn, __MODULE__, opts}

unquote(dispatches) will metaprogrammatically include do_handler functions relating to sockets. Since we are not interested in sockets at the moment, we can consider the call to __handler__ to very simply return {:plug, conn, __MODULE__, opts}.

Looking back at the Cowboy2Handler, we can see that this response maps well to the endpoint.__handler__ case of {:plug, conn, handler, opts}.

# deps/phoenix/lib/phoenix/endpoint/cowboy2_handler.ex

{:plug, conn, handler, opts} ->
  %{adapter: {@connection, req}} =
    conn
    |> handler.call(opts)
    |> maybe_send(handler)

  {:ok, req, {handler, opts}}

The handler is our own Endpoint, so handler.call is actually calling a function defined in “phoenix/endpoint.ex”, just above the do_handler definitions.

# deps/phoenix/lib/phoenix/endpoint.ex

def call(conn, opts) do
  conn = put_in conn.secret_key_base, config(:secret_key_base)
  conn = put_in conn.script_name, script_name()
  conn = Plug.Conn.put_private(conn, :phoenix_endpoint, __MODULE__)

  try do
    super(conn, opts)
  rescue
    e in Plug.Conn.WrapperError ->
      %{conn: conn, kind: kind, reason: reason, stack: stack} = e
      Phoenix.Endpoint.RenderErrors.__catch__(conn, kind, reason, stack, @phoenix_render_errors)
  catch
    kind, reason ->
      stack = System.stacktrace()
      Phoenix.Endpoint.RenderErrors.__catch__(conn, kind, reason, stack, @phoenix_render_errors)
  end
end

This function is pretty simple. It updates the conn structure with the secret key and some Phoenix-specific information, then calls super(conn, opts), delegating the rest of the work to the “parent” module, Plug.Builder. If you recall from the beginning of this post, this is exactly what we expected to happen.

Cowboy2Handler calls PhoenixInternalsWeb.Endpoint.call(conn, opts)
The Endpoint adds some data to the conn, then calls super(conn, opts), which passes control to Plug.Builder.
Plug.Builder does its thing and calls all of the plugs in the Endpoint in order.

Now, the last thing that happens in Cowboy2Handler is a call to maybe_send.

# deps/phoenix/lib/phoenix/endpoint/cowboy2_handler.ex

defp maybe_send(%Plug.Conn{state: :unset}, _plug), do: raise(Plug.Conn.NotSentError)
defp maybe_send(%Plug.Conn{state: :set} = conn, _plug), do: Plug.Conn.send_resp(conn)
defp maybe_send(%Plug.Conn{} = conn, _plug), do: conn

This will result in the response actually being sent, or, if the response was sent earlier in the pipeline, it will do nothing.

And that’s it; that’s how a Phoenix app goes from mix phx.server to actually serving responses. Of course, there’s a lot that goes on between the handler.call and maybe_send functions, but we will cover that in a later post. For now, let’s finish up by taking a step back to the beginning.

In the original __using__ macro, we had a few things going on, but we only covered the server aspect.

# deps/phoenix/lib/phoenix/endpoint.ex

defmacro __using__(opts) do
  quote do
    @behaviour Phoenix.Endpoint

    unquote(config(opts))
    unquote(pubsub())
    unquote(plug())
    unquote(server())
  end
end

We will save pubsub for another time, but there are a few things worth mentioning in config and plug.

# deps/phoenix/lib/phoenix/endpoint.ex

defp config(opts) do
  quote do
    @otp_app unquote(opts)[:otp_app] || raise "endpoint expects :otp_app to be given"
    var!(config) = Phoenix.Endpoint.Supervisor.config(@otp_app, __MODULE__)
    var!(code_reloading?) = var!(config)[:code_reloader]

    # Avoid unused variable warnings
    _ = var!(code_reloading?)

    @doc false
    def init(_key, config) do
      {:ok, config}
    end

    defoverridable init: 2
  end
end

The config function sets the @otp_app module attribute using the value passed in from our own endpoint. In this case, it’s :phoenix_internals. Then it sets a handful of variables that will be available from our endpoint at compile time. It does this with Elixir’s var! function, which is more metaprogramming.

The most interesting variable that gets set is config, whose data is fetched from the Endpoint.Supervisor. In Phoenix, your configs are actually a supervised process of their own, defined in a similar way to your server. If you recall the supervision tree diagram from above, the config worker falls on the righthand side, under “stuff we didn’t care about at the moment.” Try going back and tracing that functionality now that you have a clear idea of how it works.

The final bit of the Phoenix startup puzzle is the Endpoint’s plug function.

# deps/phoenix/lib/phoenix/endpoint.ex

defp plug() do
  quote location: :keep do
    use Plug.Builder
    
    ...snip...
  end
end

This function does a few things based on configs, but, most importantly, this is where use Plug.Builder is actually used, allowing the whole Endpoint to actually function.

End Notes

Thanks for reading my Phoenix internals post! I decided to split this into multiple parts for a few reasons. One, certainly, is because this post was getting close to 2500 words on its own. But another reason is that I wanted to provide room for feedback. I hope for this series to be helpful, and if I end up writing 10k+ words that go down avenues no one cares about, then that’s a wasted effort. So, if you’ve read through this post (or even if you just scrolled straight to the bottom), then shoot me a message with what you liked or what you want to see improved in future posts.

From the documentation, Plug is a “specification and conveniences for composable modules between web applications.” Plug internals are beyond the scope of this post, but more information can be found at https://github.com/elixir-plug/plug. ↩
For third-party Mix tasks, you’ll only find them after you’ve fetched your dependencies. ↩