Overview
In this episode of The Dashboard Effect, Brick Thompson and Landon Oaks discuss a strategic shift in how they approach data pipelines, moving away from a default preference for custom-built connectors toward a more deliberate mix that includes third-party commercial tools like Fivetran. It is a practical conversation about a decision that every data team eventually faces, and the hosts bring the kind of nuance that only comes from having built and maintained both approaches in production environments.
The takeaway is not that one approach is always better. It is that the calculus has changed, and teams that have not revisited their defaults recently may be carrying more overhead than they need to. See how Blue Margin’s Managed Data Service applies the right mix of commercial and custom pipeline tooling to keep data flowing reliably without the maintenance burden that an all-custom approach creates.
What This Episode Covers
The Shift in Strategy (1:34 – 2:32)
The team’s historical preference was to build custom connectors for granular control and cost efficiency. As APIs have grown more complex, requiring specialized coding, Python knowledge, and careful rate limit management, that preference has given way to a more pragmatic view. Commercial tools have matured to the point where they often represent a better long-term investment for standard integrations.
The Advantage of Third-Party Tools (4:03 – 4:20)
Tools like Fivetran employ dedicated teams whose entire job is to maintain and update connectors as APIs change. For an internal engineering team, every API version change is a potential pipeline break that requires immediate attention. Offloading that maintenance burden frees engineers to focus on work that is more specific to the business.
The Reality of Custom Builds (2:39 – 5:03)
Commercial tools do not cover everything. Despite offering hundreds of pre-built connectors, the hosts encounter unique or niche systems every month that require custom solutions. There is also a practical gap to consider: new feature support in commercial tools can take five to six months to arrive, making custom builds necessary when immediate data access is required.
Cost and Maintenance Considerations (5:56 – 6:30)
Commercial subscriptions carry ongoing costs, but those costs trade off against upfront development time and the long-term support overhead of maintaining custom connectors. The hosts are direct about what that support actually requires: a team that is ready to respond and fix issues as soon as they surface, which is a real and often underestimated commitment.
The Practical Conclusion (6:41 – 7:18)
The hosts advise technical teams to explore commercial pipeline tools as a default for standard integrations while maintaining the capability to build custom solutions for systems that require it. Their own work will continue to involve both, and they are candid that there is no version of this where one approach covers everything.
Who It’s For
This episode is worth your time if you are a data engineer or architect evaluating your current pipeline strategy, a technology leader weighing the build versus buy decision for data infrastructure, an engineering manager trying to reduce the maintenance burden on your team without sacrificing coverage, or anyone who has experienced the downstream pain of a broken pipeline caused by an unannounced API change.
Why It’s Worth a Listen
The build versus buy conversation in data engineering tends to get framed as a philosophical debate. This episode treats it as an operational one, which is more useful. Brick and Landon are not arguing for a position. They are describing what they have learned from doing both and adjusting their approach based on what the work actually costs over time.
The point about API complexity is worth sitting with. The assumption that custom connectors are cheaper because they avoid subscription fees does not hold up when you account for the engineering hours required to build them, maintain them, and respond when they break. For teams carrying that overhead silently, this episode offers a clear framework for reassessing whether it is still the right trade-off.
And for teams that have already moved to commercial tooling, the honest acknowledgment that custom builds are still unavoidable in practice is a useful corrective to the idea that any single tool or vendor will ever cover everything you need.
