The problem
The client is a German financial services company serving doctors and other healthcare professionals: practice founding advice, financing, and accounts, with the application handled online rather than across a branch counter.
That business model puts unusual weight on paid acquisition. A doctor deciding how to finance a practice is not a purchase that happens on impulse or in one session, and the gap between an enquiry and money actually lent is measured in weeks. So the marketing team ran four paid channels and a website, and the financing side tracked leads, opportunities, and the accounts it won and lost, in Salesforce.
The two halves never met, and that is the ordinary shape of this problem: every ad platform reports its own conversions, in its own dashboard, against its own attribution window, and each one claims the credit for the same doctor. Salesforce knew which applications actually closed and knew nothing about what any of them cost to acquire. So the question worth asking, which channel produces customers rather than merely enquiries, had no answer anywhere in the business.
Neither did a much simpler one. If enquiries stop arriving this morning, is that the market, the campaign, or a broken form.
What was built
An end-to-end analytics platform, from vendor APIs to the dashboards the business actually opens. Airbyte self-hosted on Kubernetes handles extraction and load; Snowflake is the warehouse, with schemas separated by layer and access granted per role rather than per person; dbt does the transformation in layers, ending in marts and a semantic layer; Tableau reads only from those marts.
The Salesforce side covers leads, opportunities, accounts won and lost, and users. The paid media side covers Google Ads, LinkedIn Ads and Facebook Ads alongside website activity, so spend and outcome finally sit in the same tables and can be joined.
The whole chain refreshes every fifteen minutes, incrementally where dbt allows it, so the dashboards describe the morning rather than yesterday.
The pipeline
- then
Sources
CRM and paid media, each with its own API and its own idea of a conversion.
- Salesforce
- Google Ads
- LinkedIn Ads
- Facebook Ads
- Website
- then
Airbyte
Extraction and load, self-hosted on Kubernetes in the client's own Azure tenancy.
- then
Snowflake
The warehouse. Schemas separated by layer, access granted per role.
- then
dbt
Transformation in layers, ending in marts and a semantic layer.
- staging
- intermediate
- marts
- semantic layer
Tableau
Reporting, refreshed on a fifteen-minute cycle.
Five sources, one warehouse, one set of definitions, refreshed every fifteen minutes. The shape matters in one specific way: nothing downstream of Snowflake talks to a vendor API, so a change to an attribution window at Google or Meta cannot quietly move a number in a board report.
Decisions
Self-hosted, because the client already ran its own infrastructure.
Airbyte offers a managed service and it would have been faster to stand up. It would also have been the wrong answer here. They already had Azure and an operating posture that keeps systems inside their own tenancy, which in financial services is not a matter of taste. Choosing managed would have meant arguing for an exception to a rule that exists for good reasons, on a first project, to save a few days of setup. Self-hosting on their Kubernetes cost more of my time and nothing of their trust.
Every number in a report comes from the warehouse, never from a vendor API.
Nothing downstream of Snowflake talks to Google, Meta, LinkedIn or Salesforce. This sounds like an implementation detail and is the reason the platform can be trusted: ad platforms change their attribution windows and restate history, and a dashboard querying them live would silently show a different number on Tuesday than it showed on Monday, with nothing in the system able to say why. Extracted data is a record of what the vendor said at the time it said it.
Definitions live once, in a semantic layer.
Without one, a qualified lead is defined in a Tableau calculated field, again in a dbt model, and a third time in the head of whoever built last quarter's deck. The three disagree, the disagreement surfaces in a meeting rather than in a code review, and the argument is about whose number is right instead of about what to do. Putting the definition in one place that everything reads from makes that class of meeting impossible.
Access is granted to roles, never to people.
Snowflake schemas separate raw, transformed and reporting layers, and grants attach to roles that people hold rather than to the people themselves. Doing it the other way works perfectly until the third person joins, at which point nobody can say who can see what without reading every grant individually. In financial services that is not merely untidy.
Fifteen minutes, not real time.
Streaming was available and was not worth it. Nothing in this funnel moves faster than a quarter of an hour, and the cost of real time is paid in complexity forever. Fifteen minutes was chosen because it is the point where a fall in enquiries becomes visible on the same morning rather than the next day, which turned out to matter more than anyone expected.
No machine learning, deliberately.
Attribution is exactly the problem people reach for a model to solve, and there is no model anywhere in this system. It was not a prediction problem, it was a definitions and joins problem: decide what counts as a touch, agree it with the business, and compute it the same way every time. A model would have produced an answer nobody could audit and no more true than the one a careful join gives. The interesting engineering here is data engineering, not AI, and pretending otherwise would have been the easiest way to make it worse.
What it found
Three months, one person
From nothing to a platform the business runs on.
Sole engineer across the whole chain: the Kubernetes deployment, the connectors, the warehouse and its security model, every dbt layer, and the reporting on top. Stated plainly because it is the honest scope, and because it is also the clearest limitation of how it was built. See below.
15 minutes
The gap between something happening and somebody being able to see it.
The whole chain refreshes on a fifteen-minute cycle, incrementally where dbt allows. That number is why the platform caught a problem nobody was looking for: enquiries fell, the fall was visible the same morning rather than in a weekly report, and the cause turned out to be on the website rather than in the market. Two halves of the picture in one place is what made the difference, not anything clever.
No model
The hard part was agreeing what a number means, not predicting it.
Joining four ad platforms to a CRM is usually described as an attribution problem and treated as a modelling one. Here it was a definitions problem. Once a touch, a lead and a closed application were each defined once and computed the same way every time, the question of which channel produces customers rather than enquiries answered itself.
What it does not do
- It is batch, not streaming. Fifteen minutes is the floor, and anything needing a faster answer than that is not served by this platform.
- Attribution follows a rule agreed with the business, not a multi-touch model that infers credit. That makes it auditable and arguable rather than sophisticated, which was the right trade here and would not be for everyone.
- Data flows one way. Insight arrives in Tableau and does not travel back into Salesforce, so nobody working a lead in the CRM sees what the warehouse knows about them.
- Self-hosting is a running cost paid in attention. The cluster, the connectors and their upgrades are real ongoing work, traded deliberately for control. A managed service moves that work to somebody else and the data out of the tenancy.
- The pipeline is only as stable as the APIs beneath it. When a vendor changes an endpoint or an ad account is restructured, a connector needs maintenance, and that is not an event you get to schedule.
- It was built by one person in three months, which is fast and also means the bus factor during the build was one. Everything is in code and version-controlled rather than in a notebook or a console, which is the mitigation, but it is not the same as a second engineer.
Where it stands
Delivered to production and in use. Built solo over three months in 2025, from the Azure and Kubernetes groundwork through to the dashboards the business opens.
There is no AI and no machine learning in it. It is data engineering, analytics engineering and business intelligence, which is what the problem actually required.