Overview
In this episode of The Dashboard Effect, Brick Thompson and Landon Oaks walk through a real technical problem they ran into while building a data pipeline for NetSuite and the unexpected solution that resolved it. The conversation is honest and specific, covering what failed, why it failed, and how the team found their way to something that worked by being willing to look past the obvious answer.
For any data engineer who has ever hit a wall with a modern API and assumed the problem had a modern solution, this episode offers a useful reminder that the right tool is not always the newest one. See how Blue Margin’s Managed Data Service brings the technical depth and problem-solving experience to keep data pipelines running reliably regardless of what the source system throws at them.
What This Episode Covers
The Problem: API Limitations at Scale (0:45 – 2:47)
The NetSuite REST API performed reliably for smaller datasets, but when the team attempted to pull tables containing tens of millions of records, the behavior changed entirely. The API returned no data, struggled with pagination, and eventually timed out even on requests for a single row. A tool that worked perfectly at one scale simply broke at another, with no clear warning that the threshold existed.
Diagnosing the Symptoms (2:10 – 2:36, 5:41 – 5:51)
The failure mode was inconsistent enough to be difficult to diagnose. Pagination issues, silent failures, and timeout behavior that appeared even on minimal requests made it hard to identify where exactly the problem lived. The team worked through each symptom methodically before concluding that the REST API was not going to be a viable path for this workload regardless of how it was configured.
The Solution: Pivoting to ODBC (3:08 – 4:07)
The team turned to an ODBC connector, an older technology they had initially set aside as a fallback. The results were immediate and dramatic. What the REST API could not handle reliably, ODBC handled with ease: 10,000 rows in ten seconds and full table pulls completing in under twenty minutes. The performance difference was not incremental. It was a different category of outcome entirely.
Implementation in a Cloud Environment (4:12 – 4:47, 6:23 – 6:52)
Because NetSuite is cloud-based, the team deployed a virtual machine in Azure to host the ODBC driver, effectively bridging the gap between the cloud ERP and their data lake. The architecture is not complicated, but it required a willingness to introduce a component that felt older and less elegant than the REST-based approach they had started with.
Key Takeaways (3:08 – 3:26, 6:52 – 7:06)
The broader lesson the hosts draw from the experience is straightforward: when a modern API hits a performance wall with large-scale data, legacy connectors like ODBC deserve serious consideration rather than dismissal. Landon is direct about the mindset required: when a primary solution fails, the job is to keep looking, stay flexible, and remain open to answers that do not fit the expected pattern.
Who It’s For
This episode is worth your time if you are a data engineer building or troubleshooting pipelines against NetSuite or similar cloud ERP systems, a solutions architect evaluating connector options for high-volume data extraction workloads, a technical lead who has hit unexplained failures with REST APIs at scale and wants a reference point for what might be causing them, or anyone responsible for building reliable data pipelines who values hearing how experienced teams work through problems that do not have obvious solutions.
Why It’s Worth a Listen
Most technical content presents solutions after the fact, with the messiness of the diagnostic process edited out. This episode keeps the messiness in, which is what makes it useful. Brick and Landon describe the failure in enough detail that engineers encountering similar symptoms will recognize them, and they are candid about the fact that the solution they landed on was not the one they went looking for.
The point about legacy technology is worth taking seriously. There is a tendency in data engineering to treat older connectors as something to be replaced rather than something to be evaluated on its merits. ODBC is not glamorous, but in this case it was faster, more reliable, and easier to implement than the modern alternative. That outcome does not make REST APIs less valuable in general. It makes a clear case for choosing tools based on what the workload actually requires rather than what is currently fashionable.
For teams building against large-scale ERP data sources, this episode is a practical reference that could save significant time and frustration on the next project where the obvious approach stops working and the path forward is not immediately clear.
