Data engineering salon. News and interesting reads about the world of data.
We built S3 on the design principals that we called out when we launched the service in 2006, and every time we review a design for a new feature or microservice in S3, we go back to these same principles.
How simple structured data trumps clever machine learning.
How to monitor and maintain Data Quality to make sure the data meets certain standards for specific business use-cases
I never thought I would be comparing my work in data engineering to the great Mike Myers but Ogres and Data Warehouses have a lot in common. Both are misunderstood by most and both can save the day when called upon.
The core problem is that there’s no central repository for defining a metric.
Data serving is a special business case, which demands real-time low-latency on small queries, the ability to scale and withstand abrupt peak loads, and high concurrency.
I mean, as a user, I can set up a static website in AWS, but it takes 45 steps in the console and 12 of them are highly confusing if you never did it before.
Data persistence for people who hate database servers.
Why your software should be auto-deployed within 15 minutes after you merge it, with no manual gates. This is the key to high performing teams and high-quality software.