Problem

I faced a challenge in obtaining cryptocurrency data from Binance Simple Earn platform. The platform had thousands of data points that I needed to extract, transform and load into a csv file for further analysis. However, the process was tedious and time-consuming, which made it difficult to obtain accurate data in a timely manner.

To solve this problem, I decided to create a web crawler in python using selenium that would automate the process of extracting the data from Binance Simple Earn platform and loading it into a csv file.

Solution

I created a web crawler in python using selenium that was able to extract, transform and load cryptocurrency data from Binance Simple Earn platform into a csv file. The crawler was able to obtain all cryptocurrencies that you could stake, the different yields, the duration of staking/lock-up period and lastly the availability of each lock-up slot.

The plan was to run the python script serverlessly through AWS Lambda, store the CSV via AWS S3 and query and visualise the data/create an analytics dashboard with Amazon Athena Quicksights.

Improvement/Maintenance

Once the script was ready and functional, I went back and refactored my code, to increase execution speed and for easy maintenance. However, Binance have halted the Simple Earn platform, staking services are no longer available. As a result, the web crawler is inactive.

Impact

Improved accuracy : My web crawler provided more accurate data than manual extraction methods.
Increased productivity & efficiency: The automation of the extraction process saved time and resources.

Automation is an addiction, to improve everything

Usman

- Python Developer
- Developed the web crawler using selenium and python.
- Implemented data extraction, transformation, and loading into a csv file.
- AWS Developer
- Deployed the python script serverlessly through AWS Lambda.
- Stored the CSV via AWS S3 and queried and visualised the data/create an analytics dashboard with Amazon Athena + Quicksights.

Automating Crypto Data Extraction with Python