Problem

At Storyblocks, our data teams were using various data tools, resulting in overspending and confusion among stakeholders. The lack of a standardized data infrastructure workflow led to difficulties in establishing a single source of truth. This resulted in an increase in costs due to the use of multiple tools and a decrease in confidence in the accuracy of the data produced.

The challenge was to streamline the data infrastructure by implementing a common data warehouse, ELT process, and visualization tool to reduce costs and establish a single source of truth.

Solution

We identified all existing data tools, including data warehouses (Redshift, Databricks), transformation processes (Redshift, dbt), and visualization tools (Periscope, Databricks, Google Analytics). We evaluated each tool's functionality, impact, development time, support cost, and adoption to determine the components of the modern data stack.

We collaborated with teams to select Databricks as the central data warehouse, dbt as the ELT process, and Mode as the new BI tool. We developed a migration plan to transition essential tables and processes from old infrastructure to new data stack. We organized business definitions and logic while preparing for model creation in Databricks. Lastly we focused on migrating existing dashboards to Mode.

Impact

Reduced costs associated with multiple tools
Increased confidence in the accuracy of produced data
"Quote of how incredible of an impact this had."

Team

Team Member #1 - Data Engineer
- Evaluated the functionality, impact, development time, support cost, and adoption of each tool to determine the components of the modern data stack.
- Collaborated with teams to select Databricks as the central data warehouse, dbt as the ELT process, and Mode as the new BI tool.
Team Member #2 - Embedded Analyst
- Developed a migration plan to transition essential tables and processes from old infrastructure to new data stack.
- Focused on migrating existing dashboards to Mode while organizing business definitions and logic in preparation for model creation in Databricks.

Streamlining Data Infrastructure at Storyblocks