Problem

Zeta is a Banking-Tech Startup, providing Banking-Platform-as-service. According to GDPR and data laws, all the data stored is segregated based on the region. Our data scientists and analysts needed a platform to run their algorithms and queries over these exclusive transactional payment data.

Being the first Data Engineer hire, I was asked to come up with a system to ensure they can run ad-hoc on-demand queries as well as scheduled jobs in a compliant fashion for any and all the tenants (geography-enterprise specific customer cohort).

Solution

The first two weeks were spent discovering, and understanding the functional needs of the platform.

For a month, the tools were researched and played around with to understand to get a better understanding of their setup and internals.

The next two months were spent iteratively building the core platform while slowly releasing it to the Data Science team for testing for their scheduled jobs.

Tech-Stack

Base Programming Language: Python

Open-source Data Storage/Compute Frameworks:

Apache Spark
Postgres DB
Apache Livy
JupyterHub

Deployment & Monitoring: Docker, Kubernetes, Graphana, Elastic Search

Scheduler: Azkaban

Cloud Services Used: AWS EMR, AWS S3

Impact

An in-house low-cost product, with core functionality similar to that of a Data Bricks Platform, was built, which is GDPR compliant and later was one of the products offerings audited to onboard the largest retail bank to Zeta platform.

Zemer - Data Analytics on Demand