The Phoenix Router
Note: This series is using Phoenix 1.4.1
This is the second in a multi-part series of blog posts about Phoenix internals. You can find the first post here.
This post is about the Phoenix Router. Specifically, it covers everything that happens between your Endpoint execution and the execution of your Controller action.
High Level Overview
In the last post we saw how
Plug.Builderis ultimately invoked, and we know that it will call every plug in the Endpoint in order. If we take a look at the Endpoint, at the very bottom, we can see the last plug that gets called is actually our Router.# lib/phoenix_internals_web/endpoint.ex plug PhoenixInternalsWeb.RouterLet’s take a look at our generated Router and consider what this can tell us about how the routing functionality works.
# lib/phoenix_internals_web/router.ex defmodule PhoenixInternalsWeb.Router do use PhoenixInternalsWeb, :router pipeline :browser do plug :accepts, ["html"] plug :fetch_session plug :fetch_flash plug :protect_from_forgery plug :put_secure_browser_headers end pipeline :api do plug :accepts, ["json"] end scope "/", PhoenixInternalsWeb do pipe_through :browser get "/", PageController, :index end # Other scopes may use custom stacks. # scope "/api", PhoenixInternalsWeb do # pipe_through :api # end endWe don’t see an
initorcallfunction in there, which are required forplugconformance, so those are presumably included via theusemacro at the top of the module.There are also a few keywords we might not recognize if we’ve never looked at Phoenix before —
scope,pipeline,pipe_through,get— but, knowing Elixir, we can safely assume that these are macros, also imported viause PhoenixInternalsWeb, :router.If you haven’t read the Phoenix Router documentation, consider doing so (https://hexdocs.pm/phoenix/routing.html), but I’ll give a quick refresher here as well.
Basically, pipelines are a set of plugs that act as middleware, routes are defined with special macros like “get” and “post”, and scopes bind routes and pipelines to a namespace. What this means is that when a request matches a route, the connection is processed by each pipeline in the current scope, and is then dispatched to the controller.
With our understanding of Endpoint functionality, and with a quick look at the generated code, we can assume the Router works like this:
- The Endpoint calls
Router.call(conn, opts).- The
callfunction matches a connection with a route definition.- The connection is processed by pipelines in the route’s scope.
- Finally, the connection is dispatched to an appropriate controller.
Now that we have some idea of what we should be seeing, let’s take a look at the internals.
Router Internals
Our first assumption was that the necessary Plug functions were included with
use PhoenixInternalsWeb, :router. If you’ve done any work with Phoenix, you’ll have noticed this module referenced at the top of your routers, controllers, and views. And, if you’ve created controllers and views without generators before, it’s likely that you’ve forgotten tousethe module at some point and run into errors. That makes sense because this is actually where all of Phoenix’s helpers are defined, used, or imported. As the documentation puts it, this is "[t]he entrypoint for defining your web interface, such as controllers, views, channels and so on."Let’s take a look at this module.
# lib/phoenix_internals_web.ex defmodule PhoenixInternalsWeb do def controller do quote do use Phoenix.Controller, namespace: PhoenixInternalsWeb import Plug.Conn import PhoenixInternalsWeb.Gettext alias PhoenixInternalsWeb.Router.Helpers, as: Routes end end def view do quote do use Phoenix.View, root: "lib/phoenix_internals_web/templates", namespace: PhoenixInternalsWeb # Import convenience functions from controllers import Phoenix.Controller, only: [get_flash: 1, get_flash: 2, view_module: 1] # Use all HTML functionality (forms, tags, etc) use Phoenix.HTML import PhoenixInternalsWeb.ErrorHelpers import PhoenixInternalsWeb.Gettext alias PhoenixInternalsWeb.Router.Helpers, as: Routes end end def router do quote do use Phoenix.Router import Plug.Conn import Phoenix.Controller end end def channel do quote do use Phoenix.Channel import PhoenixInternalsWeb.Gettext end end @doc """ When used, dispatch to the appropriate controller/view/etc. """ defmacro __using__(which) when is_atom(which) do apply(__MODULE__, which, []) end endDefined at the bottom of the module is the
__using__macro, where we can see that ouruseargument is ultimately used to decide which functionality to import. In our case, we supplied:routerwhich results in a call toPhoenixInternalsWeb.router(). Let’s take a look at the router function.# lib/phoenix_internals_web.ex def router do quote do use Phoenix.Router import Plug.Conn import Phoenix.Controller end endWe can see that it’s pretty simple. It imports
Plug.ConnandPhoenix.Controllerfor easy access to helper functions, and itusesPhoenix.Router. ThePhoenix.Routerseems like the obvious place for our plug functions to be defined, so we’ll take a look at it next.Since we are
useing the module, the first thing to look for is the__using__macro.# deps/phoenix/lib/phoenix/router.ex defmacro __using__(_) do quote do unquote(prelude()) unquote(defs()) unquote(match_dispatch()) end endThe
Phoenix.Routermodule is following a pattern we’ve seen throughout the codebase, where functionality is included byunquote-ing the return of various functions.To find the Plug functions we are looking for, we can search through each
unquoted function in turn, or simply cmd-fcallto find that it is located withinmatch_dispatch.# deps/phoenix/lib/phoenix/router.ex defp match_dispatch() do quote location: :keep do @behaviour Plug @doc """ Callback required by Plug that initializes the router for serving web requests. """ def init(opts) do opts end @doc """ Callback invoked by Plug on every request. """ def call(conn, _opts) do conn |> prepare() |> __match_route__(conn.method, Enum.map(conn.path_info, &URI.decode/1), conn.host) |> Phoenix.Router.__call__() end defoverridable [init: 1, call: 2] end endTaking a look at
call, the general flow is pretty simple. Theconnis prepared in some way, then the route is found via__match_route__, and the output from__match_route__is passed toRouter.__call__. Presumably, that final__call__function is what actually dispatches theconnto the appropriate controller.Let’s take a look at each of these functions to see exactly what is happening.
prepare/1# deps/phoenix/lib/phoenix/router.ex defp prepare(conn) do update_in conn.private, &(&1 |> Map.put(:phoenix_router, __MODULE__) |> Map.put(__MODULE__, {conn.script_name, @phoenix_forwards})) endThe
preparefunction adds some keys to theprivatemap in theConnstructure. These are used further on in the request lifecycle, but aren’t particularly interesting right now.
__match_route__/4The
__match_route__function is where things start to get more interesting. You’ll find it near the bottom ofbuild_matchin aquoteblock.# deps/phoenix/lib/phoenix/router.ex quote line: route.line do unquote(pipe_definition) @doc false def __match_route__(var!(conn), unquote(verb_match), unquote(path), unquote(host)) do {unquote(prepare), &unquote(Macro.var(pipe_name, __MODULE__))/1, unquote(dispatch)} end endWhen this code is compiled it will generate functions like
__match_route__(conn, "GET", ["path", id], _). This is pretty standard Elixir pattern-matching, and it’s easy to see how the match/dispatch functionality probably works. What we are missing now is how these routes are actually being defined. Let’s take a look at that.If we widen our view a bit, we’ll see that this quoted
__match_route__/4function is inbuild_match/2. Searching for thebuild_match/2call location shows that it is called within the__before_compile__macro.# deps/phoenix/lib/phoenix/router.ex defmacro __before_compile__(env) do routes = env.module |> Module.get_attribute(:phoenix_routes) |> Enum.reverse routes_with_exprs = Enum.map(routes, &{&1, Route.exprs(&1)}) Helpers.define(env, routes_with_exprs) {matches, _} = Enum.map_reduce(routes_with_exprs, %{}, &build_match/2) ... endHere, it looks like routes are being fetched from the
:phoenix_routesmodule attribute, then getting processed byRoute.exprs/1, and finallymap_reduced through ourbuild_match/2function. The interesting bit here is that routes are stored and fetched from a module attribute. Since this all happens compile-time, it’s a little difficult to introspect, but we can still test that out.To do that, let’s add
@phoenix_routesto our own router, then run the application.$ iex -S mix phx.server Erlang/OTP 22 [erts-10.4.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace] Compiling 2 files (.ex) == Compilation error in file lib/phoenix_internals_web/router.ex == ** (FunctionClauseError) no function clause matching in Phoenix.Router.Route.build_path_and_binding/1 The following arguments were given to Phoenix.Router.Route.build_path_and_binding/1: # 1 1 Attempted function clauses (showing 1 out of 1): defp build_path_and_binding(%Phoenix.Router.Route{path: path} = route) (phoenix) lib/phoenix/router/route.ex:78: Phoenix.Router.Route.build_path_and_binding/1 (phoenix) lib/phoenix/router/route.ex:63: Phoenix.Router.Route.exprs/1 (phoenix) lib/phoenix/router.ex:321: anonymous fn/1 in Phoenix.Router."MACRO-__before_compile__"/2 (elixir) lib/enum.ex:1336: Enum."-map/2-lists^map/1-0-"/2 (phoenix) expanding macro: Phoenix.Router.__before_compile__/1 lib/phoenix_internals_web/router.ex:1: PhoenixInternalsWeb.Router (module) (elixir) lib/kernel/parallel_compiler.ex:229: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7By setting our own invalid module attribute, we’ve caused the compilation to fail. Looking at the stack trace, we can see that it’s failed somewhere in the
Route.exprs/1processing. This is, perhaps, interesting, but not super helpful.Anyway, now that we know routes are stored in a module attribute, let’s take a look at where that happens.
# deps/phoenix/lib/phoenix/router.ex defp prelude() do quote do Module.register_attribute __MODULE__, :phoenix_routes, accumulate: true @phoenix_forwards %{} import Phoenix.Router # TODO v2: No longer automatically import dependencies import Plug.Conn import Phoenix.Controller # Set up initial scope @phoenix_pipeline nil Phoenix.Router.Scope.init(__MODULE__) @before_compile unquote(__MODULE__) end endThis
preludefunction is called from our initial__using__macro, and it actually sets up a number of attributes for our router module. For now, the bit we care about isModule.register_attribute __MODULE__, :phoenix_routes, accumulate: true. This registers the:phoenix_routesattribute, and indicates that repeated calls to@phoenix_routeswill accumulate instead of overwriting the previous value. We’ll find those calls within theadd_route/6function.# deps/phoenix/lib/phoenix/router.ex defp add_route(kind, verb, path, plug, plug_opts, options) do quote do @phoenix_routes Scope.route( __ENV__.line, __ENV__.module, unquote(kind), unquote(verb), unquote(path), unquote(plug), unquote(plug_opts), unquote(options) ) end endThis function is called from a number of macros (hence the
quoted return), which are defined just above.# deps/phoenix/lib/phoenix/router.ex for verb <- @http_methods do @doc """ Generates a route to handle a #{verb} request to the given path. """ defmacro unquote(verb)(path, plug, plug_opts, options \\ []) do add_route(:match, unquote(verb), path, plug, plug_opts, options) end endThis little block of code, just above
add_route, defines all of the macros you are probably familiar with in Phoenix routes. When your default router containsget "/", PageController, :index, this is the macro that is being called. In this instance, that will result in a call toadd_route(:match, :get, "/", PageController, :index, []).Every time we call one of these macros,
add_routeis called, and the return value ofScope.routeis added to our@phoenix_routesmodule attribute.Let’s move on to
Scope.route. From theadd_routefunction, we can see that it takes all of the values passed in our originalget/post/etc macro calls, as well as some line and module metadata.# deps/phoenix/lib/phoenix/router/scope.ex @doc """ Builds a route based on the top of the stack. """ def route(line, module, kind, verb, path, plug, plug_opts, opts) do path = validate_path(path) private = Keyword.get(opts, :private, %{}) assigns = Keyword.get(opts, :assigns, %{}) as = Keyword.get(opts, :as, Phoenix.Naming.resource_name(plug, "Controller")) {path, host, alias, as, pipes, private, assigns} = join(module, path, plug, as, private, assigns) Phoenix.Router.Route.build(line, kind, verb, path, host, alias, plug_opts, as, pipes, private, assigns) endAt a high level, we can see that this function fetches and validates various route details, such as the path, pipes, assigns, etc, then passes this information to the
Route.buildfunction. Although this is mostly straightforward, there is a little weirdness with scopes, which we will circle back to later.The
Route.buildfunction is very simple, and just returns aRoutestructure.# deps/phoenix/lib/phoenix/router/route.ex def build(line, kind, verb, path, host, plug, opts, helper, pipe_through, private, assigns) when is_atom(verb) and (is_binary(host) or is_nil(host)) and is_atom(plug) and (is_binary(helper) or is_nil(helper)) and is_list(pipe_through) and is_map(private) and is_map(assigns) and kind in [:match, :forward] do %Route{kind: kind, verb: verb, path: path, host: host, private: private, plug: plug, opts: opts, helper: helper, pipe_through: pipe_through, assigns: assigns, line: line} endTaking a step back, we can get a big-picture view of what happens when we define routes.
- We add a route with
get "/", PageController, :index.- This is a macro, which calls
add_route.- In turn,
add_routecalls@phoenix_routes Scope.route(...).Scope.routereturns aRoutestruct with all of the appropriate pipes, paths, etc.- This route is accumulated in
@phoenix_routesbecause it was registered with theacccumulate: trueoption.- The
__before_compile__macro is called, which fetches all of the routes withModule.get_attribute(:phoenix_routes).- The routes are processed, and then passed to the
build_matchfunction, which in turn defines the__match_route__functions.There are still a few open questions that we will be clearing up shortly, but hopefully this makes sense and you’re starting to see how all of these pieces fit together. If not, take a moment to poke through the code we’ve been reviewing, and try to get yourself oriented.
Earlier I mentioned that the
Scope.routefunction had a little weirdness. Well, that has to do with how pipes are fetched. Most information (path, method, etc) is passed directly to the route builder functions, but how exactly are routes accessing data about pipes?If we look back at the
routefunction above, we’ll see that pipes are fetched via ajoinfunction.# deps/phoenix/lib/phoenix/router/scope.ex defp join(module, path, alias, as, private, assigns) do stack = get_stack(module) {join_path(stack, path), find_host(stack), join_alias(stack, alias), join_as(stack, as), join_pipe_through(stack), join_private(stack, private), join_assigns(stack, assigns)} endThere are a number of references to a “stack” in this function, including when pipes are fetched. If we follow along with the
get_stackfunction, we’ll see that this is another instance of module attribute (ab)use. Theget_stackfunction fetches the stack from the router module withModule.get_attribute(module, :phoenix_router_scopes). This module attribute is a stack ofScopestructs, which can be seen in the scope initialization.# deps/phoenix/lib/phoenix/router/scope.ex @doc """ Initializes the scope. """ def init(module) do Module.put_attribute(module, @stack, [%Scope{}]) Module.put_attribute(module, @pipes, MapSet.new) endTo understand how this works, and why it’s implemented as a stack, let’s look back at the default router.
scope "/", PhoenixInternalsWeb do pipe_through :browser get "/", PageController, :index endInternally, this uses the
scopemacro, which creates aScopestructure, and pushes it to the stack. Then, all routes that are defined in the body of the macro can fetch the appropriate scope fields, such as path information, or pipes. Let’s verify that programmatically.First, we’ll add an
inspectto our scope.scope "/", PhoenixInternalsWeb do pipe_through :browser IO.inspect(Module.get_attribute(__MODULE__, :phoenix_router_scopes)) get "/", PageController, :index endNow, restart the Phoenix application and note the compile-time output.
[ %Phoenix.Router.Scope{ alias: "Elixir.PhoenixInternalsWeb", as: nil, assigns: %{}, host: nil, path: [], pipes: [:browser], private: %{} }, %Phoenix.Router.Scope{ alias: nil, as: nil, assigns: %{}, host: nil, path: nil, pipes: [], private: %{} } ]Our scope at the top of the stack is the one we defined ourselves. We can see the module information and the list of pipes, which will be accessible to any of the route macros defined within the scope.
Let’s add a nested scope, and two more
inspectcalls. One within the nested scope, and one after.scope "/", PhoenixInternalsWeb do pipe_through :browser IO.inspect(Module.get_attribute(__MODULE__, :phoenix_router_scopes)) get "/", PageController, :index scope "/nested", PhoenixInternalsWeb do IO.inspect(Module.get_attribute(__MODULE__, :phoenix_router_scopes)) get "/nested", PageController, :index end IO.inspect(Module.get_attribute(__MODULE__, :phoenix_router_scopes)) endNow, rebuilding the application will give us the following output:
[ %Phoenix.Router.Scope{ alias: "Elixir.PhoenixInternalsWeb", as: nil, assigns: %{}, host: nil, path: [], pipes: [:browser], private: %{} }, %Phoenix.Router.Scope{ alias: nil, as: nil, assigns: %{}, host: nil, path: nil, pipes: [], private: %{} } ] [ %Phoenix.Router.Scope{ alias: "Elixir.PhoenixInternalsWeb", as: nil, assigns: %{}, host: nil, path: ["nested"], pipes: [], private: %{} }, %Phoenix.Router.Scope{ alias: "Elixir.PhoenixInternalsWeb", as: nil, assigns: %{}, host: nil, path: [], pipes: [:browser], private: %{} }, %Phoenix.Router.Scope{ alias: nil, as: nil, assigns: %{}, host: nil, path: nil, pipes: [], private: %{} } ] [ %Phoenix.Router.Scope{ alias: "Elixir.PhoenixInternalsWeb", as: nil, assigns: %{}, host: nil, path: [], pipes: [:browser], private: %{} }, %Phoenix.Router.Scope{ alias: nil, as: nil, assigns: %{}, host: nil, path: nil, pipes: [], private: %{} } ]Note that the first stack has stayed the same. However, the second set, which corresponds to the nested scope, has a third
Scopestructure. As we can see, the “path” has been updated . Once the scope ends, theScopestructure is popped from the stack, and the final inspected stack once again contains only twoScopes.We can see this push/pop functionality in code by taking a look at the
scopemacro.# deps/phoenix/lib/phoenix/router.ex defmacro scope(path, alias, options, do: context) do options = quote do unquote(options) |> Keyword.put(:path, unquote(path)) |> Keyword.put(:alias, unquote(alias)) end do_scope(options, context) end defp do_scope(options, context) do quote do Scope.push(__MODULE__, unquote(options)) try do unquote(context) after Scope.pop(__MODULE__) end end endLet’s look back at the final steps of route building.
# deps/phoenix/lib/phoenix/router.ex defmacro __before_compile__(env) do routes = env.module |> Module.get_attribute(:phoenix_routes) |> Enum.reverse routes_with_exprs = Enum.map(routes, &{&1, Route.exprs(&1)}) Helpers.define(env, routes_with_exprs) {matches, _} = Enum.map_reduce(routes_with_exprs, %{}, &build_match/2) ... endThe routes are fetched from the module, then they are processed by
Route.exprs/1.# deps/phoenix/lib/phoenix/router/route.ex def exprs(route) do {path, binding} = build_path_and_binding(route) %{ path: path, host: build_host(route.host), verb_match: verb_match(route.verb), binding: binding, prepare: build_prepare(route, binding), dispatch: build_dispatch(route) } endThis function sets up and generates a lot of the code that will be used further on, but doesn’t make a lot of sense out of context. The best way to understand it is to examine the output. To do so, we can update our router…
# Our router scope "/", PhoenixInternalsWeb do pipe_through :browser get "/foo/:id", PageController, :index end…and add some inspection statements after the
routes_with_exprsassignment in the__before_compile__macro.# deps/phoenix/lib/phoenix/router.ex IO.inspect(routes_with_exprs) {_, expr} = hd(routes_with_exprs) Macro.to_string(expr.prepare) |> IO.puts()Now we run
mix deps.compile phoenix && mix compileand view the logs. Theroutes_with_exprsshould look something like this:{% raw %} [ { %Phoenix.Router.Route{ assigns: %{}, helper: "page", host: nil, kind: :match, line: 19, opts: :index, path: "/foo/:id", pipe_through: [:browser], plug: PhoenixInternalsWeb.PageController, private: %{}, verb: :get }, %{ binding: [{"id", {:id, [], nil}}], dispatch: {PhoenixInternalsWeb.PageController, :index}, host: {:_, [], Phoenix.Router.Route}, path: ["foo", {:id, [], nil}], prepare: {:__block__, [], [ {:=, [], [{:path_params, [], :conn}, {:%{}, [], [{"id", {:id, [], nil}}]}]}, {:=, [], [ {:%{}, [], [params: {:params, [], :conn}]}, {:var!, [context: Phoenix.Router.Route, import: Kernel], [{:conn, [], Phoenix.Router.Route}]} ]}, {:%{}, [], [ {:|, [], [ {:var!, [context: Phoenix.Router.Route, import: Kernel], [{:conn, [], Phoenix.Router.Route}]}, [ params: {{:., [], [{:__aliases__, [alias: false], [:Map]}, :merge]}, [], [{:params, [], :conn}, {:path_params, [], :conn}]}, path_params: {:path_params, [], :conn} ] ]} ]} ]}, verb_match: "GET" }} ] {% endraw %}This output is a tuple of the route, which we are already familiar with, and the return value of
Route.exprs. TheRoute.exprsreturn value, in particular thepreparekey, looks a little complicated, but these are just the values that will go into creating our__match_route__functions. We can see that from the output ofMacro.to_string(expr.prepare) |> IO.puts().( path_params = %{"id" => id} %{params: params} = var!(conn) %{var!(conn) | params: Map.merge(params, path_params), path_params: path_params} )Now, we can add another inspection line to our router, just after
{matches, _} = Enum.map_reduce(routes_with_exprs, %{}, &build_match/2).# deps/phoenix/lib/phoenix/router.ex Macro.to_string(matches) |> IO.puts()We can run this again by compiling our deps and our app (
mix deps.compile phoenix && mix compile). Hopefully, the output here is quite clear.[( defp(__pipe_through0__(conn)) do conn = Plug.Conn.put_private(conn, :phoenix_pipelines, [:browser]) case(browser(conn, [])) do %Plug.Conn{halted: true} = conn -> nil conn %Plug.Conn{} = conn -> conn other -> raise("expected browser/2 to return a Plug.Conn, all plugs must receive a connection (conn) and return a connection" <> ", got: #{inspect(other)}") end end @doc(false) def(__match_route__(var!(conn), "GET", ["foo", id], _)) do {( path_params = %{"id" => id} %{params: params} = var!(conn) %{var!(conn) | params: Map.merge(params, path_params), path_params: path_params} ), &__pipe_through0__/1, {PhoenixInternalsWeb.PageController, :index}} end )]Finally, we see the actual code generated with our router. Let’s look back at our
callfunction from the very beginning to see how it fits together.# deps/phoenix/lib/phoenix/router.ex def call(conn, _opts) do conn |> prepare() |> __match_route__(conn.method, Enum.map(conn.path_info, &URI.decode/1), conn.host) |> Phoenix.Router.__call__() endThe
connisprepared, then is passed to__match_route__. If it pattern matches against a__match_route__function like we saw above, the__match_route__function returns a tuple containing aconnwith updated path parameters, apipe_throughfunction capture, and a module/action controller tuple. Finally, that value is passed toPhoenix.Router.__call__.
Phoenix.Router.__call__/1The final function executed in the Router is perhaps its most straightforward.
# deps/phoenix/lib/phoenix/router.ex def __call__({conn, pipeline, {plug, opts}}) do case pipeline.(conn) do %Plug.Conn{halted: true} = halted_conn -> halted_conn %Plug.Conn{} = piped_conn -> try do plug.call(piped_conn, plug.init(opts)) rescue e in Plug.Conn.WrapperError -> Plug.Conn.WrapperError.reraise(e) catch :error, reason -> Plug.Conn.WrapperError.reraise(piped_conn, :error, reason, System.stacktrace()) end end endTaking the output from the
__match_route__, this function executes the pipeline, then, unless the connection is halted, it callsplug.call(piped_conn, plug.init(opts)). As you may have noticed, the “plug” variable in this case is the controller, which conforms to thePlugspec. So that is where theconnis finally dispatched.Wrapping Up
We’ve now covered everything from
mix phx.serverto the request being dispatched to an appropriate controller. This ended up being another post of 2k+ words. Sorry about that! If you made it all the way to the end, I hope you got something out of it. If you did, @ me on Twitter :)