april theses / blog posts tech, elixir September 23, 2023

Configuring Elixir's ExAws with Req and Storj

I pulled ExAws into an Elixir project recently. The app is Relinker. I intend to use cloud object storage for OpenGraph preview images for websites.

I've used ExAws before in other projects at work, but I wasn't the one who had to do the setup. Along the way, I found that configuring ExAws was not easy to get right at first.


I wanted two unrelated things:

  • I wanted to configure the HTTP client to use Req instead of the default Hackney, since I already had Req listed as a dependency.
  • Instead of S3, I wanted to connect to Storj which boasts an S3-compatible API because the free tier for Storj (25gb storage, 25gb/month egress) is really attractive for my pet project.


Here's where I landed after much trial:

config :ex_aws,
  http_client: ExAws.Request.Req,
  access_key_id: [{:system, "STORJ_ACCESS_KEY"}, :instance_role],
  secret_access_key: [{:system, "STORJ_SECRET_KEY"}, :instance_role]

config :ex_aws, :s3,
  scheme: "https://",
  host: "gateway.storjshare.io"

As a library consumer...

I think that documentation for how to configure ex_aws could be improved. In this vein, I submitted a tiny PR to improve the docs for configuring the HTTP client. But more is needed.

Here's what I learned through trial-and-error:

  • The above config cannot be runtime config. It must be build-time. In other words, put it in (e.g.) config.exs, not runtime.exs
  • You need to keep those lists of tuples and atoms in the access key fields. They're actually meaningful.
  • Putting those two ideas together: ExAws has a concept of refreshable config, which change at runtime. But that doesn't mean runtime config; ExAws uses the configuration to understand when and how to access environment variables.

The :system and :instance_role stuff that you see above is deep in the library's config-parsing and env-var grabbing logic. But, these concepts are currently necessary for developers to interact with, all the way in userland config. In other words, it's really a public API (at the moment).

This wasn't apparent to me at all while approaching ExAws for the first time. Now that I've done it, it's making more sense, but I still think that it could be made more clear. Since the docs are open-source, I will take a deeper look in the near future.


Just to be clear, I'm not raising these as criticisms of community library maintainers, which I know is a totally thankless job.

I am simply documenting my experience, in case it is useful for myself or others.

Andy Tran

Written by Andy Tran. All thoughts posted here are mine, and are not indicative of positions held by any organization or group I affiliate with.