To learn about this wonderful probablilistic data structure called Bloom filters and things that redis provides to support this really fast and low memory way of finding data.

Outline

Bloom filters are probabilistic data structures that use very low memory to allow you to find if data does not existing. It gives false positives, but cannot give false negatives. This means that it can be used to query for existance of some stuff for sure. The structure also allows that you can tradeoff space to probability of false positives.
I also go over how we used it to store infomration about a large amount of video data( about 100 million) in a small 1G redis box.
(This was a ruby implementation and I can show some snippets of this code) also there will be a javascript version based on redback(which I contributed, but not used in production)

Speaker bio

Sreekanth. Works with Activesphere as a developer. Interested in distributed systems , Ruby and javascript.

Links

• Bloom Filters
• [Bloomfilter in redback, contributed by sreeix] (https://github.com/chriso/redback)