Shadow AI Spend in Engineering Teams

Shadow AI spend is what happens when engineering teams adopt AI tools faster than Finance and Engineering leadership can build the visibility infrastructure to track them. It is not deliberate concealment — it is the natural result of individual teams solving their immediate productivity problems with the tools available to them, while the shared cost accounting layer lags six to twelve months behind adoption.

By the time someone in Finance raises a question about the cloud bill, three things have typically happened: the AI spend is real and growing; it is mixed into a general API line item with no team-level breakdown; and Engineering leadership cannot tell Finance which team drove which portion of the cost — because there is no per-team attribution in place. The problem is not that teams spent money. The problem is the absence of a system that records who spent it and for what purpose.

This article covers the four layers that close that gap: detection, per-team attribution, gateway-level metering architecture, and the governance gates that prevent shadow spend from accumulating in future quarters.

Why shadow AI spend happens

The adoption pattern is consistent across organisations. An individual engineer discovers that an AI coding assistant reduces their context-switching overhead on a debugging task. They share it with two colleagues. Those two share it with their team leads. Within a quarter, three or four teams are running AI-assisted workflows — some using company-provided seat licences, some using personal API keys expensed through corporate cards, some using shared API keys created informally for "just trying this out."

Each of these access patterns generates spend with different accounting footprints. Seat licences appear as SaaS subscriptions, typically in Finance's software budget. API calls through shared or personal keys appear in the cloud bill under the account that holds the API credentials — which may be a personal developer account or a shared team AWS or GCP account that Finance does not monitor with the same granularity as the primary cloud account. Expensed corporate card transactions appear in T&E or procurement depending on the company's classification rules.

None of these three footprints naturally join in a single view. An engineering manager trying to answer "how much did my team spend on AI tooling last quarter" typically has to query at least three systems — the cloud bill, the SaaS subscription management tool, and the corporate card expense reports — and do a manual join. In practice this does not happen until Finance asks the question, at which point the answer takes days to construct and is still an estimate.

The second driver is the absence of an approval gate on new AI tool adoption. When a developer spins up an API key for a new model provider, there is typically no workflow that requires a budget owner to acknowledge the cost profile of what they are enabling. A developer who has used Claude for single prompts at $0.003 per call may not have a mental model of what 50-iteration agentic workflows cost on the same API. Approval gates for new tool adoption are not about slowing teams down — they are about ensuring someone with budget context has seen the cost profile before the workflow runs in production at scale.

Detection patterns

The first step is building an inventory of where AI spend is currently happening. This requires looking in more places than most organisations expect.

Cloud billing tags. Every major cloud provider supports resource tags or labels that can be applied to API calls via the request header or SDK configuration. AWS Bedrock supports tagging via the user-agent header and via resource tags on provisioned throughput. Anthropic's API supports a X-Amzn-Trace-Id-style header approach and logs cost at the key level (with detailed billing available in the Console). The first detection step is pulling the last 90 days of API cost by key ID from every provider in use. This tells you the volume of spend per key. It does not yet tell you which team or workflow generated each key's spend — that requires the attribution layer below.

Per-team API key inventory. The second detection step is building a map of which API keys exist, who created them, and which team they nominally belong to. Most organisations discover more keys than expected — developers create keys for experiments, forget to rotate or delete them, and the experiment turns into production usage. A key inventory audit against each provider's API (Anthropic's console, AWS IAM, GitHub Copilot seats dashboard) produces the list. Cross-reference each key against the cost data from step one. Keys with nonzero spend and no owner metadata are the shadow spend candidates.

Per-workflow attribution headers. For internal tooling that calls LLM APIs programmatically, the most granular detection mechanism is injecting attribution headers into every API call at the SDK configuration level. Anthropic's API accepts arbitrary metadata in the metadata parameter of the messages endpoint; this field accepts key-value pairs and is stored with the request for cost attribution in the usage dashboard. Set metadata["team_id"], metadata["workflow_id"], and metadata["task_type"] on every call from internal tooling. The cost dashboard then shows spend by team, workflow, and task type — which is the granularity Finance actually needs to answer their question.

Expense report grep. Do a one-time sweep of the last six months of corporate card transactions for known AI vendor names: "Anthropic," "OpenAI," "Cursor," "Augment," "GitHub Copilot." This surfaces any seat licences or personal-key charges being expensed individually rather than going through procurement. The total is usually smaller than the API-call footprint but reveals adoption signals — teams that are expensing individual tool licences will likely move to shared API keys within a quarter if they find the tool useful.

Attribution architecture

Detection tells you how much was spent. Attribution tells you by whom and for what purpose. The two architectures for building attribution are gateway-level metering and per-call tagging. Each has different implementation overhead and different coverage guarantees.

Gateway-level metering routes all LLM API calls through an internal proxy that sits between application code and the external API. Every call passes through the gateway, which records the calling service identity, the model requested, the token count from the response, and any custom metadata the calling service supplies. The gateway aggregates this into a per-team, per-workflow spend dashboard that updates in near real-time.

The coverage guarantee is strong: if traffic cannot reach the external API without going through the gateway, every call is metered. The implementation overhead is real: you need an internal HTTP proxy that is highly available (LLM API latency is already 200ms–2s; adding a gateway hop should add under 10ms), handles authentication credential rotation centrally, and can be updated with new routing rules without redeploying consuming services. Open-source options like CloudZero (cost intelligence layer, not a proxy) and LiteLLM (open-source LLM proxy) cover different parts of this problem. Evaluate against your org's reliability requirements before adopting.

Per-call tagging injects attribution data into every API call at the SDK level, without a separate gateway. The advantage is zero additional infrastructure: the metadata travels with the API call and appears in the provider's own billing exports. The coverage guarantee is weaker: it depends on all calling code following the tagging convention. Code that was written before the convention was established — or code written by a team that was not briefed — will generate unattributed spend. Retrofit is achievable but requires touching every API call site in the codebase, which is a one-time cost that scales with codebase age.

In practice, most organisations use both: a gateway for internal agentic tooling (where routing through an internal proxy is architecturally clean), and per-call tagging for edge cases where tooling connects directly. Whichever pattern you adopt, the goal is the same — every call has a team, a cost-centre, and a workflow attached at the time of the call, not reconstructed later from a billing CSV.

Org-level governance gates

Attribution tells you what happened in the past. Governance gates prevent uncontrolled accumulation in the future. Three gates cover the majority of shadow spend patterns.

Per-team monthly hard caps. Each team's AI spend gets a monthly ceiling, set by the budget owner (usually an engineering manager or director in collaboration with Finance). The ceiling is not punitive — it is a forcing function for the conversation about what value the spend is generating. When a team's spend approaches the ceiling, the gateway or billing alert fires, the budget owner reviews, and either approves an increase (logging the rationale) or investigates which workflow drove the spike. A team that regularly needs increases is a team whose AI tooling is producing value above expectations — that is the visibility signal Finance actually wants.

Approval workflows for new agent deployments. When a new agentic workflow is proposed — one that will run unattended, call the LLM API in a loop, or scale spend with data volume rather than with headcount — it goes through a lightweight review that covers: estimated token spend per run; expected run frequency; maximum per-session circuit breaker threshold; and the team owner responsible for monitoring it. This review does not need to be heavyweight. A one-page document reviewed asynchronously by the budget owner and approved before the workflow goes to production is sufficient. The goal is ensuring that no agentic workflow enters production without a cost estimate and a named owner. The budget gate and per-session circuit breaker patterns that keep individual workflows bounded are covered in the AI agent token spend control guide.

Quarterly attribution audit. At the end of each quarter, the gateway's attribution data feeds a report: spend by team, spend by workflow type, spend vs the per-team ceiling, and a list of any unattributed spend (keys with no team metadata). Unattributed spend is a finding that requires remediation before the next quarter — either the source is identified and tagged retroactively, or the key is revoked. The quarterly audit gives Finance the breakdown they need for the next budget cycle and gives Engineering leadership the data to make informed decisions about which teams and workflows are generating value at what cost.

What this looks like with an AI gateway

A gateway-based attribution system at the point where shadow spend becomes visible looks like this in practice: all internal services that call LLM APIs are configured to route through an internal gateway endpoint (for example, https://ai-gateway.internal/v1/messages instead of directly to api.anthropic.com). The gateway authenticates outbound calls using a centrally-managed API key set, so individual teams do not hold raw provider credentials — they hold a gateway token that scopes their access to their team's allowed models and monthly budget.

The gateway records every request with service identity (from the gateway token), model, input token count, output token count, cached token count, and any custom metadata the calling service passes. This data feeds a near-real-time spend dashboard. When a team's spend crosses 80% of their monthly ceiling, the gateway sends a Slack notification to the team lead and the budget owner. When spend crosses 100%, the gateway returns a structured error ({"error": "team_budget_exceeded"}) and the calling service handles it according to its own policy — either failing gracefully or escalating to the budget owner for a cap increase.

The implementation overhead for this architecture is real — typically two to four weeks of platform work to get a reliable gateway with authentication, routing, and billing export integration. The payoff is that Finance's question ("who spent the $200k?") becomes a three-second query rather than a three-day audit. For a 50-engineer org spending $150k–$300k per year on AI tooling, the attribution infrastructure pays for itself within the first billing cycle where it prevents a mispriced budget increase or surfaces a workflow that was spending 10× its expected cost.