The Phoenix Router
Note: This series is using Phoenix 1.4.1
This is the second in a multi-part series of blog posts about Phoenix internals. You can find the first post here.
This post is about the Phoenix Router. Specifically, it covers everything that happens between your Endpoint execution and the execution of your Controller action.
High Level Overview
In the last post we saw how
Plug.Builder
is ultimately invoked, and we know that it will call every plug in the Endpoint in order. If we take a look at the Endpoint, at the very bottom, we can see the last plug that gets called is actually our Router.# lib/phoenix_internals_web/endpoint.ex plug PhoenixInternalsWeb.Router
Let’s take a look at our generated Router and consider what this can tell us about how the routing functionality works.
# lib/phoenix_internals_web/router.ex defmodule PhoenixInternalsWeb.Router do use PhoenixInternalsWeb, :router pipeline :browser do plug :accepts, ["html"] plug :fetch_session plug :fetch_flash plug :protect_from_forgery plug :put_secure_browser_headers end pipeline :api do plug :accepts, ["json"] end scope "/", PhoenixInternalsWeb do pipe_through :browser get "/", PageController, :index end # Other scopes may use custom stacks. # scope "/api", PhoenixInternalsWeb do # pipe_through :api # end end
We don’t see an
init
orcall
function in there, which are required forplug
conformance, so those are presumably included via theuse
macro at the top of the module.There are also a few keywords we might not recognize if we’ve never looked at Phoenix before —
scope
,pipeline
,pipe_through
,get
— but, knowing Elixir, we can safely assume that these are macros, also imported viause PhoenixInternalsWeb, :router
.If you haven’t read the Phoenix Router documentation, consider doing so (https://hexdocs.pm/phoenix/routing.html), but I’ll give a quick refresher here as well.
Basically, pipelines are a set of plugs that act as middleware, routes are defined with special macros like “get” and “post”, and scopes bind routes and pipelines to a namespace. What this means is that when a request matches a route, the connection is processed by each pipeline in the current scope, and is then dispatched to the controller.
With our understanding of Endpoint functionality, and with a quick look at the generated code, we can assume the Router works like this:
- The Endpoint calls
Router.call(conn, opts)
.- The
call
function matches a connection with a route definition.- The connection is processed by pipelines in the route’s scope.
- Finally, the connection is dispatched to an appropriate controller.
Now that we have some idea of what we should be seeing, let’s take a look at the internals.
Router Internals
Our first assumption was that the necessary Plug functions were included with
use PhoenixInternalsWeb, :router
. If you’ve done any work with Phoenix, you’ll have noticed this module referenced at the top of your routers, controllers, and views. And, if you’ve created controllers and views without generators before, it’s likely that you’ve forgotten touse
the module at some point and run into errors. That makes sense because this is actually where all of Phoenix’s helpers are defined, used, or imported. As the documentation puts it, this is "[t]he entrypoint for defining your web interface, such as controllers, views, channels and so on."Let’s take a look at this module.
# lib/phoenix_internals_web.ex defmodule PhoenixInternalsWeb do def controller do quote do use Phoenix.Controller, namespace: PhoenixInternalsWeb import Plug.Conn import PhoenixInternalsWeb.Gettext alias PhoenixInternalsWeb.Router.Helpers, as: Routes end end def view do quote do use Phoenix.View, root: "lib/phoenix_internals_web/templates", namespace: PhoenixInternalsWeb # Import convenience functions from controllers import Phoenix.Controller, only: [get_flash: 1, get_flash: 2, view_module: 1] # Use all HTML functionality (forms, tags, etc) use Phoenix.HTML import PhoenixInternalsWeb.ErrorHelpers import PhoenixInternalsWeb.Gettext alias PhoenixInternalsWeb.Router.Helpers, as: Routes end end def router do quote do use Phoenix.Router import Plug.Conn import Phoenix.Controller end end def channel do quote do use Phoenix.Channel import PhoenixInternalsWeb.Gettext end end @doc """ When used, dispatch to the appropriate controller/view/etc. """ defmacro __using__(which) when is_atom(which) do apply(__MODULE__, which, []) end end
Defined at the bottom of the module is the
__using__
macro, where we can see that ouruse
argument is ultimately used to decide which functionality to import. In our case, we supplied:router
which results in a call toPhoenixInternalsWeb.router()
. Let’s take a look at the router function.# lib/phoenix_internals_web.ex def router do quote do use Phoenix.Router import Plug.Conn import Phoenix.Controller end end
We can see that it’s pretty simple. It imports
Plug.Conn
andPhoenix.Controller
for easy access to helper functions, and ituse
sPhoenix.Router
. ThePhoenix.Router
seems like the obvious place for our plug functions to be defined, so we’ll take a look at it next.Since we are
use
ing the module, the first thing to look for is the__using__
macro.# deps/phoenix/lib/phoenix/router.ex defmacro __using__(_) do quote do unquote(prelude()) unquote(defs()) unquote(match_dispatch()) end end
The
Phoenix.Router
module is following a pattern we’ve seen throughout the codebase, where functionality is included byunquote
-ing the return of various functions.To find the Plug functions we are looking for, we can search through each
unquote
d function in turn, or simply cmd-fcall
to find that it is located withinmatch_dispatch
.# deps/phoenix/lib/phoenix/router.ex defp match_dispatch() do quote location: :keep do @behaviour Plug @doc """ Callback required by Plug that initializes the router for serving web requests. """ def init(opts) do opts end @doc """ Callback invoked by Plug on every request. """ def call(conn, _opts) do conn |> prepare() |> __match_route__(conn.method, Enum.map(conn.path_info, &URI.decode/1), conn.host) |> Phoenix.Router.__call__() end defoverridable [init: 1, call: 2] end end
Taking a look at
call
, the general flow is pretty simple. Theconn
is prepared in some way, then the route is found via__match_route__
, and the output from__match_route__
is passed toRouter.__call__
. Presumably, that final__call__
function is what actually dispatches theconn
to the appropriate controller.Let’s take a look at each of these functions to see exactly what is happening.
prepare/1
# deps/phoenix/lib/phoenix/router.ex defp prepare(conn) do update_in conn.private, &(&1 |> Map.put(:phoenix_router, __MODULE__) |> Map.put(__MODULE__, {conn.script_name, @phoenix_forwards})) end
The
prepare
function adds some keys to theprivate
map in theConn
structure. These are used further on in the request lifecycle, but aren’t particularly interesting right now.
__match_route__/4
The
__match_route__
function is where things start to get more interesting. You’ll find it near the bottom ofbuild_match
in aquote
block.# deps/phoenix/lib/phoenix/router.ex quote line: route.line do unquote(pipe_definition) @doc false def __match_route__(var!(conn), unquote(verb_match), unquote(path), unquote(host)) do {unquote(prepare), &unquote(Macro.var(pipe_name, __MODULE__))/1, unquote(dispatch)} end end
When this code is compiled it will generate functions like
__match_route__(conn, "GET", ["path", id], _)
. This is pretty standard Elixir pattern-matching, and it’s easy to see how the match/dispatch functionality probably works. What we are missing now is how these routes are actually being defined. Let’s take a look at that.If we widen our view a bit, we’ll see that this quoted
__match_route__/4
function is inbuild_match/2
. Searching for thebuild_match/2
call location shows that it is called within the__before_compile__
macro.# deps/phoenix/lib/phoenix/router.ex defmacro __before_compile__(env) do routes = env.module |> Module.get_attribute(:phoenix_routes) |> Enum.reverse routes_with_exprs = Enum.map(routes, &{&1, Route.exprs(&1)}) Helpers.define(env, routes_with_exprs) {matches, _} = Enum.map_reduce(routes_with_exprs, %{}, &build_match/2) ... end
Here, it looks like routes are being fetched from the
:phoenix_routes
module attribute, then getting processed byRoute.exprs/1
, and finallymap_reduce
d through ourbuild_match/2
function. The interesting bit here is that routes are stored and fetched from a module attribute. Since this all happens compile-time, it’s a little difficult to introspect, but we can still test that out.To do that, let’s add
@phoenix_routes
to our own router, then run the application.$ iex -S mix phx.server Erlang/OTP 22 [erts-10.4.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace] Compiling 2 files (.ex) == Compilation error in file lib/phoenix_internals_web/router.ex == ** (FunctionClauseError) no function clause matching in Phoenix.Router.Route.build_path_and_binding/1 The following arguments were given to Phoenix.Router.Route.build_path_and_binding/1: # 1 1 Attempted function clauses (showing 1 out of 1): defp build_path_and_binding(%Phoenix.Router.Route{path: path} = route) (phoenix) lib/phoenix/router/route.ex:78: Phoenix.Router.Route.build_path_and_binding/1 (phoenix) lib/phoenix/router/route.ex:63: Phoenix.Router.Route.exprs/1 (phoenix) lib/phoenix/router.ex:321: anonymous fn/1 in Phoenix.Router."MACRO-__before_compile__"/2 (elixir) lib/enum.ex:1336: Enum."-map/2-lists^map/1-0-"/2 (phoenix) expanding macro: Phoenix.Router.__before_compile__/1 lib/phoenix_internals_web/router.ex:1: PhoenixInternalsWeb.Router (module) (elixir) lib/kernel/parallel_compiler.ex:229: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7
By setting our own invalid module attribute, we’ve caused the compilation to fail. Looking at the stack trace, we can see that it’s failed somewhere in the
Route.exprs/1
processing. This is, perhaps, interesting, but not super helpful.Anyway, now that we know routes are stored in a module attribute, let’s take a look at where that happens.
# deps/phoenix/lib/phoenix/router.ex defp prelude() do quote do Module.register_attribute __MODULE__, :phoenix_routes, accumulate: true @phoenix_forwards %{} import Phoenix.Router # TODO v2: No longer automatically import dependencies import Plug.Conn import Phoenix.Controller # Set up initial scope @phoenix_pipeline nil Phoenix.Router.Scope.init(__MODULE__) @before_compile unquote(__MODULE__) end end
This
prelude
function is called from our initial__using__
macro, and it actually sets up a number of attributes for our router module. For now, the bit we care about isModule.register_attribute __MODULE__, :phoenix_routes, accumulate: true
. This registers the:phoenix_routes
attribute, and indicates that repeated calls to@phoenix_routes
will accumulate instead of overwriting the previous value. We’ll find those calls within theadd_route/6
function.# deps/phoenix/lib/phoenix/router.ex defp add_route(kind, verb, path, plug, plug_opts, options) do quote do @phoenix_routes Scope.route( __ENV__.line, __ENV__.module, unquote(kind), unquote(verb), unquote(path), unquote(plug), unquote(plug_opts), unquote(options) ) end end
This function is called from a number of macros (hence the
quote
d return), which are defined just above.# deps/phoenix/lib/phoenix/router.ex for verb <- @http_methods do @doc """ Generates a route to handle a #{verb} request to the given path. """ defmacro unquote(verb)(path, plug, plug_opts, options \\ []) do add_route(:match, unquote(verb), path, plug, plug_opts, options) end end
This little block of code, just above
add_route
, defines all of the macros you are probably familiar with in Phoenix routes. When your default router containsget "/", PageController, :index
, this is the macro that is being called. In this instance, that will result in a call toadd_route(:match, :get, "/", PageController, :index, [])
.Every time we call one of these macros,
add_route
is called, and the return value ofScope.route
is added to our@phoenix_routes
module attribute.Let’s move on to
Scope.route
. From theadd_route
function, we can see that it takes all of the values passed in our originalget
/post
/etc macro calls, as well as some line and module metadata.# deps/phoenix/lib/phoenix/router/scope.ex @doc """ Builds a route based on the top of the stack. """ def route(line, module, kind, verb, path, plug, plug_opts, opts) do path = validate_path(path) private = Keyword.get(opts, :private, %{}) assigns = Keyword.get(opts, :assigns, %{}) as = Keyword.get(opts, :as, Phoenix.Naming.resource_name(plug, "Controller")) {path, host, alias, as, pipes, private, assigns} = join(module, path, plug, as, private, assigns) Phoenix.Router.Route.build(line, kind, verb, path, host, alias, plug_opts, as, pipes, private, assigns) end
At a high level, we can see that this function fetches and validates various route details, such as the path, pipes, assigns, etc, then passes this information to the
Route.build
function. Although this is mostly straightforward, there is a little weirdness with scopes, which we will circle back to later.The
Route.build
function is very simple, and just returns aRoute
structure.# deps/phoenix/lib/phoenix/router/route.ex def build(line, kind, verb, path, host, plug, opts, helper, pipe_through, private, assigns) when is_atom(verb) and (is_binary(host) or is_nil(host)) and is_atom(plug) and (is_binary(helper) or is_nil(helper)) and is_list(pipe_through) and is_map(private) and is_map(assigns) and kind in [:match, :forward] do %Route{kind: kind, verb: verb, path: path, host: host, private: private, plug: plug, opts: opts, helper: helper, pipe_through: pipe_through, assigns: assigns, line: line} end
Taking a step back, we can get a big-picture view of what happens when we define routes.
- We add a route with
get "/", PageController, :index
.- This is a macro, which calls
add_route
.- In turn,
add_route
calls@phoenix_routes Scope.route(...)
.Scope.route
returns aRoute
struct with all of the appropriate pipes, paths, etc.- This route is accumulated in
@phoenix_routes
because it was registered with theacccumulate: true
option.- The
__before_compile__
macro is called, which fetches all of the routes withModule.get_attribute(:phoenix_routes)
.- The routes are processed, and then passed to the
build_match
function, which in turn defines the__match_route__
functions.There are still a few open questions that we will be clearing up shortly, but hopefully this makes sense and you’re starting to see how all of these pieces fit together. If not, take a moment to poke through the code we’ve been reviewing, and try to get yourself oriented.
Earlier I mentioned that the
Scope.route
function had a little weirdness. Well, that has to do with how pipes are fetched. Most information (path, method, etc) is passed directly to the route builder functions, but how exactly are routes accessing data about pipes?If we look back at the
route
function above, we’ll see that pipes are fetched via ajoin
function.# deps/phoenix/lib/phoenix/router/scope.ex defp join(module, path, alias, as, private, assigns) do stack = get_stack(module) {join_path(stack, path), find_host(stack), join_alias(stack, alias), join_as(stack, as), join_pipe_through(stack), join_private(stack, private), join_assigns(stack, assigns)} end
There are a number of references to a “stack” in this function, including when pipes are fetched. If we follow along with the
get_stack
function, we’ll see that this is another instance of module attribute (ab)use. Theget_stack
function fetches the stack from the router module withModule.get_attribute(module, :phoenix_router_scopes)
. This module attribute is a stack ofScope
structs, which can be seen in the scope initialization.# deps/phoenix/lib/phoenix/router/scope.ex @doc """ Initializes the scope. """ def init(module) do Module.put_attribute(module, @stack, [%Scope{}]) Module.put_attribute(module, @pipes, MapSet.new) end
To understand how this works, and why it’s implemented as a stack, let’s look back at the default router.
scope "/", PhoenixInternalsWeb do pipe_through :browser get "/", PageController, :index end
Internally, this uses the
scope
macro, which creates aScope
structure, and pushes it to the stack. Then, all routes that are defined in the body of the macro can fetch the appropriate scope fields, such as path information, or pipes. Let’s verify that programmatically.First, we’ll add an
inspect
to our scope.scope "/", PhoenixInternalsWeb do pipe_through :browser IO.inspect(Module.get_attribute(__MODULE__, :phoenix_router_scopes)) get "/", PageController, :index end
Now, restart the Phoenix application and note the compile-time output.
[ %Phoenix.Router.Scope{ alias: "Elixir.PhoenixInternalsWeb", as: nil, assigns: %{}, host: nil, path: [], pipes: [:browser], private: %{} }, %Phoenix.Router.Scope{ alias: nil, as: nil, assigns: %{}, host: nil, path: nil, pipes: [], private: %{} } ]
Our scope at the top of the stack is the one we defined ourselves. We can see the module information and the list of pipes, which will be accessible to any of the route macros defined within the scope.
Let’s add a nested scope, and two more
inspect
calls. One within the nested scope, and one after.scope "/", PhoenixInternalsWeb do pipe_through :browser IO.inspect(Module.get_attribute(__MODULE__, :phoenix_router_scopes)) get "/", PageController, :index scope "/nested", PhoenixInternalsWeb do IO.inspect(Module.get_attribute(__MODULE__, :phoenix_router_scopes)) get "/nested", PageController, :index end IO.inspect(Module.get_attribute(__MODULE__, :phoenix_router_scopes)) end
Now, rebuilding the application will give us the following output:
[ %Phoenix.Router.Scope{ alias: "Elixir.PhoenixInternalsWeb", as: nil, assigns: %{}, host: nil, path: [], pipes: [:browser], private: %{} }, %Phoenix.Router.Scope{ alias: nil, as: nil, assigns: %{}, host: nil, path: nil, pipes: [], private: %{} } ] [ %Phoenix.Router.Scope{ alias: "Elixir.PhoenixInternalsWeb", as: nil, assigns: %{}, host: nil, path: ["nested"], pipes: [], private: %{} }, %Phoenix.Router.Scope{ alias: "Elixir.PhoenixInternalsWeb", as: nil, assigns: %{}, host: nil, path: [], pipes: [:browser], private: %{} }, %Phoenix.Router.Scope{ alias: nil, as: nil, assigns: %{}, host: nil, path: nil, pipes: [], private: %{} } ] [ %Phoenix.Router.Scope{ alias: "Elixir.PhoenixInternalsWeb", as: nil, assigns: %{}, host: nil, path: [], pipes: [:browser], private: %{} }, %Phoenix.Router.Scope{ alias: nil, as: nil, assigns: %{}, host: nil, path: nil, pipes: [], private: %{} } ]
Note that the first stack has stayed the same. However, the second set, which corresponds to the nested scope, has a third
Scope
structure. As we can see, the “path” has been updated . Once the scope ends, theScope
structure is popped from the stack, and the final inspected stack once again contains only twoScope
s.We can see this push/pop functionality in code by taking a look at the
scope
macro.# deps/phoenix/lib/phoenix/router.ex defmacro scope(path, alias, options, do: context) do options = quote do unquote(options) |> Keyword.put(:path, unquote(path)) |> Keyword.put(:alias, unquote(alias)) end do_scope(options, context) end defp do_scope(options, context) do quote do Scope.push(__MODULE__, unquote(options)) try do unquote(context) after Scope.pop(__MODULE__) end end end
Let’s look back at the final steps of route building.
# deps/phoenix/lib/phoenix/router.ex defmacro __before_compile__(env) do routes = env.module |> Module.get_attribute(:phoenix_routes) |> Enum.reverse routes_with_exprs = Enum.map(routes, &{&1, Route.exprs(&1)}) Helpers.define(env, routes_with_exprs) {matches, _} = Enum.map_reduce(routes_with_exprs, %{}, &build_match/2) ... end
The routes are fetched from the module, then they are processed by
Route.exprs/1
.# deps/phoenix/lib/phoenix/router/route.ex def exprs(route) do {path, binding} = build_path_and_binding(route) %{ path: path, host: build_host(route.host), verb_match: verb_match(route.verb), binding: binding, prepare: build_prepare(route, binding), dispatch: build_dispatch(route) } end
This function sets up and generates a lot of the code that will be used further on, but doesn’t make a lot of sense out of context. The best way to understand it is to examine the output. To do so, we can update our router…
# Our router scope "/", PhoenixInternalsWeb do pipe_through :browser get "/foo/:id", PageController, :index end
…and add some inspection statements after the
routes_with_exprs
assignment in the__before_compile__
macro.# deps/phoenix/lib/phoenix/router.ex IO.inspect(routes_with_exprs) {_, expr} = hd(routes_with_exprs) Macro.to_string(expr.prepare) |> IO.puts()
Now we run
mix deps.compile phoenix && mix compile
and view the logs. Theroutes_with_exprs
should look something like this:{% raw %} [ { %Phoenix.Router.Route{ assigns: %{}, helper: "page", host: nil, kind: :match, line: 19, opts: :index, path: "/foo/:id", pipe_through: [:browser], plug: PhoenixInternalsWeb.PageController, private: %{}, verb: :get }, %{ binding: [{"id", {:id, [], nil}}], dispatch: {PhoenixInternalsWeb.PageController, :index}, host: {:_, [], Phoenix.Router.Route}, path: ["foo", {:id, [], nil}], prepare: {:__block__, [], [ {:=, [], [{:path_params, [], :conn}, {:%{}, [], [{"id", {:id, [], nil}}]}]}, {:=, [], [ {:%{}, [], [params: {:params, [], :conn}]}, {:var!, [context: Phoenix.Router.Route, import: Kernel], [{:conn, [], Phoenix.Router.Route}]} ]}, {:%{}, [], [ {:|, [], [ {:var!, [context: Phoenix.Router.Route, import: Kernel], [{:conn, [], Phoenix.Router.Route}]}, [ params: {{:., [], [{:__aliases__, [alias: false], [:Map]}, :merge]}, [], [{:params, [], :conn}, {:path_params, [], :conn}]}, path_params: {:path_params, [], :conn} ] ]} ]} ]}, verb_match: "GET" }} ] {% endraw %}
This output is a tuple of the route, which we are already familiar with, and the return value of
Route.exprs
. TheRoute.exprs
return value, in particular theprepare
key, looks a little complicated, but these are just the values that will go into creating our__match_route__
functions. We can see that from the output ofMacro.to_string(expr.prepare) |> IO.puts()
.( path_params = %{"id" => id} %{params: params} = var!(conn) %{var!(conn) | params: Map.merge(params, path_params), path_params: path_params} )
Now, we can add another inspection line to our router, just after
{matches, _} = Enum.map_reduce(routes_with_exprs, %{}, &build_match/2)
.# deps/phoenix/lib/phoenix/router.ex Macro.to_string(matches) |> IO.puts()
We can run this again by compiling our deps and our app (
mix deps.compile phoenix && mix compile
). Hopefully, the output here is quite clear.[( defp(__pipe_through0__(conn)) do conn = Plug.Conn.put_private(conn, :phoenix_pipelines, [:browser]) case(browser(conn, [])) do %Plug.Conn{halted: true} = conn -> nil conn %Plug.Conn{} = conn -> conn other -> raise("expected browser/2 to return a Plug.Conn, all plugs must receive a connection (conn) and return a connection" <> ", got: #{inspect(other)}") end end @doc(false) def(__match_route__(var!(conn), "GET", ["foo", id], _)) do {( path_params = %{"id" => id} %{params: params} = var!(conn) %{var!(conn) | params: Map.merge(params, path_params), path_params: path_params} ), &__pipe_through0__/1, {PhoenixInternalsWeb.PageController, :index}} end )]
Finally, we see the actual code generated with our router. Let’s look back at our
call
function from the very beginning to see how it fits together.# deps/phoenix/lib/phoenix/router.ex def call(conn, _opts) do conn |> prepare() |> __match_route__(conn.method, Enum.map(conn.path_info, &URI.decode/1), conn.host) |> Phoenix.Router.__call__() end
The
conn
isprepare
d, then is passed to__match_route__
. If it pattern matches against a__match_route__
function like we saw above, the__match_route__
function returns a tuple containing aconn
with updated path parameters, apipe_through
function capture, and a module/action controller tuple. Finally, that value is passed toPhoenix.Router.__call__
.
Phoenix.Router.__call__/1
The final function executed in the Router is perhaps its most straightforward.
# deps/phoenix/lib/phoenix/router.ex def __call__({conn, pipeline, {plug, opts}}) do case pipeline.(conn) do %Plug.Conn{halted: true} = halted_conn -> halted_conn %Plug.Conn{} = piped_conn -> try do plug.call(piped_conn, plug.init(opts)) rescue e in Plug.Conn.WrapperError -> Plug.Conn.WrapperError.reraise(e) catch :error, reason -> Plug.Conn.WrapperError.reraise(piped_conn, :error, reason, System.stacktrace()) end end end
Taking the output from the
__match_route__
, this function executes the pipeline, then, unless the connection is halted, it callsplug.call(piped_conn, plug.init(opts))
. As you may have noticed, the “plug” variable in this case is the controller, which conforms to thePlug
spec. So that is where theconn
is finally dispatched.Wrapping Up
We’ve now covered everything from
mix phx.server
to the request being dispatched to an appropriate controller. This ended up being another post of 2k+ words. Sorry about that! If you made it all the way to the end, I hope you got something out of it. If you did, @ me on Twitter :)