Site Disabled

Platform 

Note, that these were just what was mentioned in the talk, it’s not a comprehensive list.

• HDFS

• S3

• Nagios

• The real time distributed messaging system bitly uses is Nsq.

• Bitly uses hostpool to manage a pool of hosts.

• The end of the talk is cut off, but it’s mentioned bitly uses quite a few different databases. 

Lessons Learned 

• Knowledge is power. The more you understand the properties of the things you are working with the better decisions you can make and the more efficiently you can work. Efficiency means you can make bigger, faster systems at a much lower cost.

• Build stuff that deals with leaky abstractions. If you are using a layer of abstraction that hides the underlying distributed nature of the system it will eventually bubble through. Your code has to understand and deal with any leaks.

• Put it all on one box if you can. If you don’t need a distributed system then don’t build one. They are complicated and expensive to build.

• In a Service Oriented Architecture failure means reduced functionality instead of going down.

• SOA + queues + async messaging is really powerful. This approach isolates components, lets work happen concurrently, lets boxes fail independently, while still having components be easy to reason about.

• Use synchronous requests when speed and consistency are paramount. Give users an error rather than a slow or wrong answer.

• Event style messages are better than command style messages. They allow for better isolation between systems and naturally support multiple consumers. Helps keep services focussed and not worried about anything beyond what the service is supposed to do.

• Annotate rather than filter. Filters at the producer level bake in assumptions about what the downstream services care about. Example is public vs private links. Filtering private links out of the stream means that services interested in private links won’t get the links they need. Instead, annotate an event with the fact that a link is either private or public and let services operate only on the kind of events they care about.

• Services should play nice with each other. Use back pressure to prevent a service from being overloaded and route around failed services.

• If there isn’t a Nagios check on something it’s almost certainly broken. You just don’t know it yet.

• Tools should expose information to humans. Get the right information to the right people at the right time.

  word-break: break-word;