Analytics is a governance decision with a query engine attached

AWS’s analytics decision guide sorts its services into streaming, batch and ETL, warehousing and lakes, and business intelligence. That is a sensible taxonomy and it maps onto the three-way tension everyone recognises: freshness against cost against who can write the query.

You can have any two. Second-level freshness means something running continuously, which costs. Cheap means idle between questions, which means batch. Self-service SQL means the questions change without a deployment, which constrains the engines you can use. Every analytics architecture is a position on that triangle, and most of the argument in a design review is people standing at different corners.

That argument is worth having. It is also not usually the one that decides.

The constraint that decides

Who may see which rows, and where is that enforced?

In a regulated estate this is the question that removes options, and it is absent from most comparisons because it is not a property of a query engine. It is a property of the whole path from source system to dashboard, and every hop on that path is an opportunity to lose it.

Consider what usually happens. The source system has real permissions — carefully built, argued over, audited. A pipeline extracts from it. The extraction runs as a service account with broad read access, because it has to read everything. It writes files to a lake. Those files have no idea what the source permissions were.

Nobody decided to discard the access model. It was discarded by the shape of the pipeline, in a change nobody reviewed as a security change, because it was a data engineering ticket.

Now an analyst queries the lake and sees rows the source system would never have shown them. The dashboard is correct. The pipeline is correct. The estate has a finding.

Three places to enforce it, and the honest costs

In the source, by never extracting what a reader may not see. Cleanest and usually impractical: it means one pipeline per audience.

In the catalogue layer, where table, column and row rules are declared once and enforced for every engine that reads through it. This is the design most worth aiming at — and its sharp edge is the phrase through it. The files underneath are still open files in a bucket. Read the bucket directly, with a role that permits it, and the rules are simply not there. That is not a defect; it is the architecture. But it means “the lake is governed” is only true while every reader goes through the front door, and proving that is an access-review exercise, not an architecture diagram.

In the BI tool. Common, and the weakest, because it protects one consumer and nothing else. The moment somebody connects a notebook, the control is gone.

Pick one, write down which, and then go and check what a service account can reach — because that account is the one that will be used to prove the control does not hold.

Run the sieve

Which of these are true for you?

• An answer must be current to within secondsA dashboard someone acts on, not one they read on Monday.
• The people asking the questions write SQL, not codeNo Spark job, no notebook, no deployment to change a question.
• Different analysts must see different rowsEnforced by the platform, not by everyone remembering to add a WHERE clause.
• It is queried a few times a dayReal work, rarely — the pattern that punishes anything provisioned.
• The data must stay readable without this serviceOpen files in our own bucket, not rows inside an engine.

Still standing8 of 8

• Amazon Athena

  SQL against files where they already sit, with nothing running between questions.

• Amazon Redshift

  A warehouse that rewards being asked hard questions repeatedly, with the statistics and caching to match.

• Amazon EMR

  Spark, Hive and friends when the transformation is genuinely a program rather than a query.

• AWS Glue

  Managed ETL plus the catalogue that tells everything else what the files mean.

• Amazon Kinesis Data Streams

  The ordered, replayable log that makes second-level freshness possible at all.

• Managed Service for Apache Flink

  Continuous computation over a stream — windows, joins and aggregates that update as events arrive.

• AWS Lake Formation

  Where table, column and row permissions are declared once and enforced for every engine that reads through it.

• Amazon QuickSight

  The dashboard layer, and usually the only part of this list a stakeholder ever sees.

Our reading, not a specification. Service behaviour changes — check any elimination that decides something against the service page before you design around it.

The pairing worth trying: an answer must be current to within seconds with it is queried a few times a day. Those two are close to contradictory, and teams ask for both routinely. Seeing them eliminate almost everything is a faster way to have that conversation than another meeting.

Freshness is usually a business question in disguise

When a stakeholder asks for real-time, the useful reply is not an architecture. It is: what decision will you make differently if this is five seconds old rather than an hour old?

Sometimes there is a genuine answer — a fraud check, a queue that needs rebalancing, an alert that reaches somebody who can act. Then build the streaming path, and accept that something now runs continuously and somebody owns its checkpoints and its restarts.

Frequently the answer is that the report is read on Monday morning. That is a batch pipeline, and the honest version costs a fraction of the streaming one and breaks in ways that are visible on Monday morning rather than at 04:00 on a Sunday.

Getting to that answer is not a technical conversation, which is exactly why it tends not to happen in the technical design where the engine is chosen.

Next, and last in this series: machine learning, where the catalogue is longest, the guide dates fastest, and the constraint is where the data goes.