Search was the first problem I started exploring from a product perspective. I was lucky to start my career in academic libraries, which were especially notable for their poor search experiences.
This mock and prototypes I built (2005) led me to develop more complete search and browsing tools that won awards (2006) and kicked off a minor revolution (2007) through improved usability (2006) and meaningfully changing how people interact with libraries and archives (2009). I later used many of the same approaches commercially to drive engagement and revenue in subscription media (2014).
DBaaS was one of the most requested upstack services from customers at Joyent. Prototyping our solution in public (and as open source) earned us substantial goodwill and positive PR. The result was the Autopilot Pattern and a number of implementations for completely automated operations.
My team contributed to the following work.
- MySQL on Autopilot with code and technical demo.
- ELK stack on Autopilot with code
- Consul, etcd, Redis, and MongoDB
This approach to automation started with a prototype I built using Couchbase for a keynote demo at Couchbase Connect. That prototype led to the development of ContainerPilot. MySQL was the first formal implementation of a database in the Autopilot Pattern.
Less public experience includes scaled Postgres production, lessons from failure, and an unconventional attempt to eke out more performance. We can also talk about performance and failure testing Cassandra in preparation for multiple >1,000 node deployments. (Weird fact: ARP cache exhaustion can significantly affect performance at scale.)
Samsung’s acquisition of Joyent allowed me to scale the team, infrastructure, and services to host applications serving billions of mobile devices and storing exabytes of data. Scaling our object storage solution that fast was a herculean effort for everybody involved. I published my review of literature and prior art that I did while exploring how others had approached scaling and cost containment. (Extra, here’s my estimate of S3’s size.)
My writing on persistent storage patterns for cloud applications is also relevant here:
The conventional wisdom is that containers are great for stateless applications, but inappropriate for stateful apps with persistent data. If this is true, it’s not because the technology isn’t up to it, it’s because the patterns for managing persistent data and stateful applications are not commonly understood. The challenge isn’t so much about how to persist state, but how to do so without compromising the agility and automation that we love about containerization in the first place.
Any reasonably scaled data store needs to address discovery between nodes. In the debate about whether this was an infrastructure concern or application concern, I argued discovery and connection management are core concerns that affect the app’s failure modes and consistency behavior:
Passive discovery patterns are those that separate the application from these decisions, leaving the application passive in both the choice of what back ends to connect to, and passive in resolving failures that may result. Active discovery patterns move those decisions into the application so it can have an active role in choosing the backend and working around failures it may encounter.