Letting AI agents look inside production BEAM systems, safely

A note on how this was made: Porthole was written with AI coding agents. I set the direction, made the design and security decisions, and reviewed and tested the results, including on our own production apps, while agents wrote most of the code, tests and documentation. Fittingly, AI also helped me edit this article: the ideas and opinions are mine, and it helped with grammar, clarity and structure.

Introduction

While attending ElixirConf, it was impossible not to talk about AI and the tsunami of changes the industry is going through. In his keynote on the second day, Chris McCord asked the audience how many were using agents and, if I remember correctly, around 70% were.

The week after, at Rails World, DHH's keynote ("Look at all the things I'm not doing") acknowledged that 37signals is no longer using Ruby for a lot of its development, a huge statement from the people who created Rails. I started my career as a Rails developer, so this hit close to home, and I think he is partially right: Ruby is a great language for humans, but the cost of that is performance. That tradeoff has always made sense economically, and the balance now seems to be shifting.

Adding to that feeling, a lot of the conversations at ElixirConf were about one question: "will programming languages even matter?". I have taken part in this kind of conversation countless times in the last year, and this is where I have finally landed: syntax won't matter, but the runtime will.

I firmly believe that the BEAM has a lot of advantages in the AI age, and one of them is its great introspection tools. If you have ever had to troubleshoot a production outage on a BEAM system, you have surely seen how much of the system's state is exposed. You can run commands from the console, across nodes, on live systems and get invaluable information about what's happening right now: processes, supervisors, ETS tables, ports and more. On top of that, the ecosystem has developed great tools to get even more information, for example live tracing with recon, among others. I think this is a great differentiator compared to other languages and tools, and that its value will compound in the AI age if used properly.

The problem

You may imagine where this is going: the BEAM has great introspection tools, so let's give agents access to query the runtime. Well, not so fast, cowboy! The BEAM offers an easy way to connect to its nodes through the cookie mechanism. This is great: there is an easy and secure way to connect to the nodes remotely and fetch information. It means we could share the cookie with an agent and have it fetch the information it needs to troubleshoot what's going on.

The problem is that the cookie gives access to everything: not only reading information from the running nodes, but also modifying the running system, which can be dangerous. You may try to make agents behave by being explicit in prompts or context files, but that's not enough.

In an experiment, I gave an agent a demo app with planted problems to investigate, and access through a remote shell and the cookie. Along the way, it copied a whole mailbox (hundreds of thousands of messages) and a whole ETS table (around a million rows) into its shell, read the internal state of several processes, ran application code to measure latency, and turned on tracing on a live process. Its report ended with "I didn't change anything". On a small demo, that's harmless, but on a production node in the middle of an incident, copying a million-row table may turn an incident into an outage. (The full eval.)

I think that if we want the BEAM to succeed as a platform in this new age, we need to add the necessary guardrails to make it safe for agents to interact with our running systems.

Introducing Porthole

Porthole is an open-source tool that allows agents to query live BEAM systems securely using SQL. Its name comes from a window you can look through but cannot pass through, which is exactly what we want to give our agents for now, until we are confident enough to give them more access.

How does it work?

Through its CLI, Porthole exposes your production system to your agents over MCP. Behind the scenes, its commands deploy a sidecar to your infrastructure (Fly.io and Kubernetes are supported): a separate, hidden node that holds the cookie and discovers the nodes of your app. The sidecar exposes an MCP endpoint, protected by a token, that the agent connects to in order to query the system. The sidecar has no public interface to the internet: you connect to it through a tunnel, using an access token.

For every query:

Nothing runs between queries, and nothing is stored.

The tables

Every table has a node column, so you can query the whole cluster with one query.

The window

A single snapshot tells you how much, not how fast. Many of the BEAM's numbers only ever grow: a process's reductions count all the work it has done since it started, so a process with two billion reductions might be busy right now, or might have been ticking along quietly for months. During an incident, what you want to know is what is changing now.

That's what the window_ms option is for. With window_ms: 5000, Porthole collects the tables twice, five seconds apart, matches each row with its earlier self (the same process, ETS table, port or node), and adds a _delta column for each counter: reductions_delta, memory_delta and message_queue_len_delta for processes, size_delta for ETS tables, bytes in and out for ports. Nothing runs on the nodes in between, and the deltas are ordinary columns, so they can be sorted, filtered and joined like any other:

-- Which processes are burning CPU right now? (window_ms: 5000)
SELECT coalesce(registered_name, initial_call) AS process, reductions_delta
FROM processes ORDER BY reductions_delta DESC LIMIT 3;

process                | reductions_delta
-----------------------+-----------------
Shop.Inventory.Sync    | 697781204
Shop.Search.Indexer    | 183653
Shop.Orders.EventRelay | 98993

Erlang developers will recognize the idea from recon's proc_window, which ranks processes by how much an attribute changed over a time window. Porthole takes the same approach and extends it: it works for every table, across every node of the cluster, and the results land in SQL. So "which ETS table grows fastest, and who owns it?" or "which supervisor keeps restarting its children?" becomes a single query.

Why SQL and SQLite

One of the reasons for choosing SQL is that agents already know it very well, and it's very flexible. By exposing the system as SQL tables, the agent can usually answer a specific question with a single query, for example:

The deadlock one shows the point best. In the demo app, two processes call each other and wait forever:

WITH RECURSIVE chain(node, start, pid, path, depth) AS (
  SELECT node, pid, waiting_on, coalesce(registered_name, pid), 1
  FROM processes WHERE waiting_on IS NOT NULL
  UNION ALL
  SELECT c.node, c.start, p.waiting_on,
         c.path || ' -> ' || coalesce(p.registered_name, p.pid), c.depth + 1
  FROM chain c JOIN processes p ON p.node = c.node AND p.pid = c.pid
  WHERE p.waiting_on IS NOT NULL AND c.pid <> c.start AND c.depth < 20
)
SELECT path AS cycle FROM chain WHERE pid = start;

cycle
----------------------------
Shop.Cart -> Shop.Promotions
Shop.Promotions -> Shop.Cart

No hand-written tool would have offered "find cycles in who waits for whom", but an agent can write it. With custom, hand-written tools, every new question means a new tool or new filter options. Besides, hand-rolling would mean building the whole querying and filtering layer from scratch, which is a lot of work. SQLite is mature and fits this use case very well: it supports an authorizer, which Porthole uses to deny writes, and a cap on its memory.

Because of how Porthole is designed, this layer is replaceable: the collectors work without SQL, so we could change the querying layer down the road without touching them.

Why it's safe

The most important aspect of Porthole's design is that the cookie to your production system lives in the sidecar deployed to your infrastructure. The agent can only talk to the sidecar over MCP, and there is no way for it to get hold of the cookie, which means it can't run anything directly on your nodes.

This holds as long as Porthole is the agent's only way into production. An agent that can also use your fly or kubectl credentials, for example from the same laptop, can go around it: Porthole makes its own tool read-only, it doesn't make the rest of your access safe.

The only code that runs on your production nodes is Porthole's own collection functions, which are fixed when Porthole is built, and a build-time check guarantees they only call OTP and only read.

Besides that, Porthole never reads ETS contents, process state or mailboxes, and the SQL queries the agent can run go through an authorizer: writes are denied.

Another fair concern is how collecting data impacts the nodes running production code. To be as gentle as possible with production, collection runs at low priority, with a deadline enforced on the node itself, because :erpc doesn't stop remote work once the caller gives up. Rows, bytes, time, sampling windows, SQLite's memory, concurrent queries and queries per client are all capped, and every cut is reported in the result. In my tests on CPU-saturated nodes, it had no measurable effect on app latency. I tried normal priority, but ultimately rejected it because it caused 180–300 ms stalls in app processes.

Another important advantage of being separate from the app is that an overloaded or failing app doesn't take Porthole down with it during an outage, which is exactly when you need it.

Access to Porthole is restricted and audited. Each agent gets its own access token, and every query is logged with the token that sent it, so you can review how agents troubleshot the situations they faced. The sidecar is reachable only through a tunnel (fly proxy, kubectl port-forward), the request's Origin is checked, and request size is capped.

No changes to your app to try it

Porthole sends its read-only collection code over Erlang distribution, and the node runs it with :erl_eval, which ships with OTP. Any Elixir app on OTP 27+ with node names of the form name@host (the Phoenix default) works, and no dependency or redeploy are required.

You might be thinking: I may not need to change my code, but I do need to change my release process to use a fixed cookie, so the sidecar can be configured with it. To make things even easier, Porthole reads the cookie from your running release and passes it to the sidecar as a secret, without ever printing it or saving it on your machine.

To try it out, you only need the fly CLI (or kubectl) logged in to your production account. It takes a couple of minutes:

# Install Porthole as a Mix archive
mix archive.install hex porthole

# Deploy the sidecar to your app
mix porthole.fly.up my-app

# When you're done, remove everything
mix porthole.fly.down my-app

fly.up deploys the sidecar from a prebuilt public image, checks that it sees your nodes, and prints two commands for you to run: the tunnel, and the one that connects your agent. fly.down only touches the sidecar that up created, and asks you to type its name first as a confirmation. Kubernetes works the same way.

Has Porthole been tried for real?

Yes! I have tried it on Elixir Observer and JustPolls (a small SaaS product we have built). Both are Fly apps, and everything worked perfectly.

I have also tried the Kubernetes path with an example app on a real AWS cluster, and it works, but I haven't tried it on a real production app yet.

Now we are looking into trying it with some of our customers, in particular the Plurality team, who are excited to try it!

What's next & conclusions

Tracing. The information Porthole exposes today is very useful for a general assessment of a system, but it's a set of snapshots: it shows a mailbox holding 40,000 messages, not who is sending them, or a busy GenServer, not which call is slow. The clear next step is tracing. It's another great thing about the BEAM: you can trace function calls, messages and crashes on a live system, and tools like dbg and recon make it part of everyday debugging when you need to dig deeper into where a problem is. Empowering agents with tracing would be a big step, but it comes with its own challenges. A careless trace can flood a node or be left running long after the investigation ends. Safe tracing means time and event limits enforced on the node, automatic cleanup, and recording metadata rather than application data.

Agents as first responders. Another direction I imagine things are going is agents as incident first responders. Picture an incident at 3 AM: your monitoring tool sends an alert to Slack and PagerDuty, and at the same time an agent spins up with read-only access to production. It starts querying and tracing at the moment it's most valuable, while the incident is still happening, before a restart or a deploy wipes out the evidence. By the time the on-call engineer opens the alert, a first diagnosis is waiting in the thread. I think Porthole is a great fit for making this vision real. It comes with a fair set of challenges, starting with how to run these agents safely. It will probably mean running them inside the production infrastructure, with access to the Porthole sidecar and nothing else. That's complex, but given the potential benefits, it's definitely worth exploring.

I'm not sure what software development will look like in this AI age, but I'm convinced that agents getting better information about running production systems is part of it, and that languages whose runtimes allow introspection will have an advantage. Who knows, maybe future runtimes will be designed with safe introspection for agents from the ground up.

If you found this interesting, give it a spin: it only takes a couple of minutes to try it on your own app, and one command to remove it. Let me know what you think, and which of the next steps would help your team more.

Links: