![]() API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. This means that our API had to be extremely stable.įor this, we leveraged Amazon’s API Gateway and AWS Lambda services. To reliably support this, we needed to ensure that every single event sent to us from our customers made it into the system. Basically, we needed an HTTP API that could handle high velocity event ingestion, and a way to process these events. Now that we had a way for our users to set things up, we needed a reliable endpoint to get all this data into our system. For example, users could set their application up to forward events to Google Analytics, Mixpanel, MailChimp, Keen.io, and so on. ![]() This web app allows users to register their own applications, and configure the various integrations they want to use. We threw together an embarrassingly ugly web application using Meteor and MongoDB as the user interface to our SaaS offering. To do that, we had to start iterating, and quick. We were a bit strapped on humans at the time, so it was important for us to leverage the resources we did have to prove we could provide real value by the time Demo Day rolled around in November. Amazon and Google’s cloud platforms are known as Infrastructure as a Service (IaaS), and these services take a lot of low-level management out of the picture and let users focus on their specific applications. AngelPad is the top ranked accelerator in the world and comes with some pretty nice perks, like credits on Amazon and Google cloud platforms. These projects solved the issue of getting data out of applications, but we still had to implement the backend machinery to actually do something with it.Īs we set out to build this first system in September 2015, we were simultaneously participating in the AngelPad accelerator Fall 2015 class. As an early stage startup with very limited resources, leveraging projects like analytics.js and similar open source libraries for other platforms was a no-brainer. It turns out a nice chunk of the code required to put something like this together has already been open sourced. This would allow our users to build live dashboards however they like, using the tools they already had or experimenting with multiple tools before deciding which one they wanted. One of the first problems we tackled was getting clickstream data out of applications and into third party tools or a data warehouse in near real-time. ![]() We became Astronomer to focus on solving our customers’ data problems, but at that point, we were still a work in progress. While we did that, we uncovered new problems, and more importantly, found new opportunities to bring value to our customers. ![]() We dug in and brainstormed ways to smooth this process out. That was a big problem because, without it, the product was useless. The largest obstacle we found ourselves fighting was actually getting our customers’ user data. Switching GearsĪstronomer was born as a pivot away from another product called USERcycle, which gave customers visual insights into how their users behaved while interacting with their applications, over time. As Astronomer’s CTO, I’m going to chronicle our journey so far, from a technical perspective, as we grow our platform and home in on how to meet our users’ real needs. We exist because we believe the internet age is just a precursor to something much larger, something with the potential to push the world forward in the same ways the Agricultural and Industrial Revolutions did: the Data Revolution.Īs you can imagine, preparing for a Revolution doesn’t happen overnight. In other words, we help you organize, centralize, and clean your data through a personalized data engineering experience. Astronomer is a modern platform built to outfit organizations with a solid data infrastructure to support machine learning and analytical workloads.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |