What you gain
AWS handles the spiky, low-volume tier and its bill stays small. The high-volume tier — egress, transcode, storage — runs on capacity you have already paid for.
The application tier runs on AWS, where elasticity genuinely helps and the bill is small. Origin storage, encoding and delivery run on hardware you own, because those are the three line items that grow with every viewer. This is the pattern that changes a streaming bill the most, and it is the one most services should be on.
Split the platform by which half is expensive to rent, not by which half is technically interesting. The application tier — APIs, catalogue, console, sign-in, billing — is spiky, elastic and cheap on somebody else's compute. Absorbing a traffic spike is exactly what you are paying AWS for, and the bill for that half is small.
The other half is three meters that grow with every viewer. Encoding bills per output minute, per rendition, with no volume tier at all. Origin storage bills per gigabyte-month against a catalogue that only grows. Delivery bills per gigabyte against an audience you are trying to grow. Those are the lines that turn commercial success into a larger invoice, and they are the ones this pattern moves onto hardware you own.
The size of that is checkable rather than asserted. One full encode pass over a 1,000-hour catalogue bills $3,060 on rented transcoding and bills again in full after any ladder change; on a GPU you already own it is a scheduling decision. On the South Asian delivery card, tripling delivered volume multiplies the bill 2.89× while the blended rate improves 3.8% — so scale is not a lever here the way it is in North America.
What the split costs is the honest other half. Capacity is bought before it is needed, utilisation below plan is money already spent, and somebody has to be reachable when hardware fails. Below roughly a terabyte a month the meter is genuinely cheaper, and the calculator on the pricing page will say so.
Every tier named, every boundary drawn. Blue is capacity you own; grey is capacity you rent.
Untrusted — the public internet
No media byte is ever served from here
Every media byte leaves from here
The bytes never enter a metered path
The same pipeline provisions AWS and the metal
| Tier | Where it runs | Why there | Billing shape |
|---|---|---|---|
| Catalogue & search | AWS | Spiky, tiny payloads | Per request, negligible |
| Auth & entitlements | AWS | Must be globally reachable | Per request, negligible |
| Billing | AWS | Availability matters more than cost | Per request, negligible |
| Encoding | Your metal | GPU already owned; no per-minute meter | Capital, already spent |
| Origin storage | Your metal | Storage plus egress is the worst rented pair | Capital, already spent |
| Delivery / CDN | Your edge | Egress is the largest recurring cost in streaming | Bandwidth commit only |
| Control plane | Git | One workflow across both estates | None |
Every architecture pays for something. Here is the bill for this one.
AWS handles the spiky, low-volume tier and its bill stays small. The high-volume tier — egress, transcode, storage — runs on capacity you have already paid for.
Two estates to operate. That is only acceptable because both are declared in the same Terraform and shipped by the same pipeline, so it is one workflow, not two.
If AWS is unreachable, sign-in and catalogue degrade but playback of already-authorised sessions continues from your edge. That split is deliberate and it is tested.
You want cloud elasticity for the parts that spike and you refuse to pay metered egress for the parts that do not need to be rented.
Three zones in the direction a request travels, with the rented control plane deliberately off the media path. Every zone is marked owned or rented, because that distinction is the architecture.
Viewers
BDIX edge
Proxmox origin
AWS control plane
The hybrid pattern is one rule applied consistently: control crosses the boundary, media does not, and telemetry crosses back the other way in aggregate. Everything below is that rule expanded — what runs where, what is allowed on the wire between them, and what happens when one side is gone.
Control hostnames and media hostnames are disjoint, and that separation is enforced in DNS rather than in application code. The catalogue and sign-in APIs resolve to the AWS tier. Playlist and segment hostnames resolve to edge nodes. No response from the control plane ever contains a URL that points back at AWS for a media object, including as a redirect.
The redirect case is the one worth stating outright, because it is the usual way a hybrid design quietly stops being hybrid. A 302 issued by AWS for every segment is not a rounding error at streaming volumes: it is one extra request on the critical path for each of the thousands of segments a session consumes, it makes the control plane a hard dependency of playback, and it puts a rented service in a path you moved specifically to stop renting. So the token, not a redirect, carries the routing decision.
Segment authorisation is verified offline. The control plane signs a short-lived playback token with a private key; each edge node holds only the corresponding public key and validates locally. There is no per-segment call to AWS, no shared session store to reach across the boundary, and no cache of authorisation decisions that can go stale in an interesting way. The token is the entitlement, it is checked where the bytes are, and it expires on its own.
The consequence is worth being explicit about rather than discovering later: a viewer holding a valid token keeps watching whether or not AWS is reachable. That is a property of where verification happens, not a failover feature bolted on afterwards.
Telemetry is the one payload that travels inward, and it does not travel the way it was produced. Player beacons are posted to a collector running on the same edge node that served the segments — same hostname family, same TLS termination, no cross-boundary hop at beacon time. The collector batches, drops obvious duplicates, and writes to the observability plane on the Proxmox side.
Per-session rows stay there. What crosses into the control plane is rolled up: series by channel, by zone, by device class and by time bucket, which is what an operator dashboard actually renders. The raw events are high-cardinality and enormous; the aggregates are small and boring, and small and boring is the correct shape for anything crossing a metered boundary.
There is a second reason beyond cost, and in an incident it is the more important one. If beacons were posted to the rented tier, then the moment that tier became unreachable you would lose the measurements describing the outage — precisely the window you will be asked about afterwards. Collecting locally means the record of a control-plane failure survives the control-plane failure.
Retention follows the same split. Full-resolution events are kept on owned storage for as long as the disk budget allows, because keeping them costs nothing per query. Aggregates are what get shipped, and they are what the admin interface reads.
Placement follows workload shape, not preference. Encoding is steady, saturating and predictable: it runs at close to full utilisation for as long as there is a catalogue to process, which is the exact profile that is expensive to rent by the minute and cheap to own outright. A GPU bought once and kept busy has no meter attached to it.
Origin storage is the other half of that argument, and it is worse than it first looks, because storage and egress are billed as a pair. A library grows monotonically and is read constantly by the edge tier. Rented, both halves grow together and neither one ever goes down. Owned, the read is free and the growth is a purchase decision made once a year.
The control tier has the opposite shape. Catalogue lookups and sign-ins are small, spiky, and far more sensitive to availability than to unit cost. That tier genuinely benefits from what a public cloud sells — managed Postgres, multiple availability zones, someone else's on-call rota for the database — and its bill stays small because none of the bytes that matter pass through it.
Proxmox is what makes the owned side operable rather than merely cheap: clustered hosts, live migration for maintenance without a viewer-visible window, and local storage sitting next to the encoders instead of across a network. It is the part of the estate that behaves like infrastructure rather than like a set of servers.
The cost of all this is two estates, and it is a real cost. It is only acceptable because both are declared in one Terraform root with two providers and shipped by one pipeline. If they were two workflows, the saving would go straight back out as operational overhead.
Zoomed in from the estate diagram to a single transaction: what a player asks for, what the edge node answers with, and the one case in which anything crosses the private path to the Proxmox origin.
Step 1, Viewer: Token already in hand
The player signed in against the control tier at session start and holds a short-lived signed token. Nothing after this point consults that tier again.
Authorization: Bearer <signed, short TTL>Step 2, BDIX edge node: Master playlist
Resolved to the nearest edge node by DNS. The manifest is tiny and permanently warm, so it is answered out of memory.
GET /hls/{asset}/master.m3u8 · 200Step 3, BDIX edge node: Signature verified locally
Public-key verification against the token's claims — asset, expiry, and the zone it was issued for. No round trip leaves the node to make this decision.
verify(pubkey) · ~microsecondsStep 4, BDIX edge node: Segment lookup
Cache key includes the byte range, so a partial read does not evict or duplicate the whole object. Most requests end here.
GET /hls/{asset}/{rendition}/{n}.tsStep 5, BDIX edge node: Hit — served from NVMe
The common path. The bytes leave the edge node to a viewer on the same exchange and never touch the private path or the origin.
200 · X-Cache: HITStep 6, Proxmox origin: Miss — one coalesced fetch
Concurrent misses for the same object collapse into a single upstream request. A thousand viewers arriving at a new segment produce one origin read, not a thousand.
GET origin/{asset}/{n}.ts · Range:Step 7, Proxmox origin: Origin serves the range
MinIO answers the range from owned storage over the private path. The edge node stores it and answers the waiting requests from the copy it just wrote.
206 Partial ContentStep 8, Viewer: Beacon returns
Playback telemetry posts to the collector on the same edge node. Rolled-up series cross into the control tier later; the per-session record stays on owned storage.
POST /beacon · batchedViewer connections end at the edge node. The origin speaks only to edge nodes over the private path, which keeps its certificate surface to one internal name.
The origin has no concept of a subscriber. It serves objects to nodes that are already authorised to ask, and entitlement is decided one tier out.
Client addresses stop at the edge. What reaches the origin is an edge node identity, which is also why origin logs are useless for audience analysis and are not used for it.
Coalescing at the edge means concurrency is absorbed before it becomes origin load. Origin request rate tracks catalogue churn, not audience size.
The value of drawing the boundary in a specific place is that failure becomes predictable. Each row is a deliberate consequence of where verification and storage sit, not a mitigation added afterwards.
| What is unreachable | Playback in progress | New sessions | Admin and billing |
|---|---|---|---|
| AWS control tier | Continues to token expiry | Sign-in fails | Unavailable |
| A single edge node | Re-routed to the next node | Unaffected | Unaffected |
| Proxmox origin | Continues from cache | Cache hits only | Unaffected |
| The private path | Continues from cache | Cache hits only | Unaffected |
| Local exchange peering | Falls back to transit | Falls back to transit | Unaffected |
No. Media hostnames resolve to edge nodes, origin storage runs on owned hardware, and the control plane never returns a URL pointing at itself for a media object — not as a body, not as a redirect. The only thing the control tier contributes to playback is a signed token issued once at session start.
The control plane signs a short-lived playback token with a private key. Each edge node holds the public key and verifies the signature and claims locally. Authorisation is therefore a local computation on the node serving the bytes, which is why playback survives a control-plane outage until the token expires.
To a collector on the same edge node that served the segments, then into the observability plane on the Proxmox side. Only rolled-up series — by channel, zone, device class and time bucket — cross into the control tier. Per-session events stay on owned storage, which is also why the record of an outage survives the outage.
Encoding, origin storage and egress are steady, high-volume workloads, which is the profile that is expensive to rent and cheap to own. The control tier is spiky, small and availability-sensitive, which is the profile a public cloud is genuinely good at. Placement follows workload shape rather than a preference for either.
One extra thing to reason about, which is real. It is contained by declaring both sides in a single Terraform root with two providers and shipping them from one pipeline, so it stays one workflow rather than two. If it were two workflows, the operational overhead would consume the saving.
Most evaluations end up comparing all three. They are all delivered, and none of them is the default answer.
Send what you have — hardware, cloud account, current bill, launch date. The answer is usually one of these three, and occasionally it is that you should change nothing.