The Fifth Elephant 2017

On data engineering and application of ML in diverse domains


Interestingness of interestingness measures

Submitted by Simrat Hanspal (@simrathanspal) on Sunday, 30 April 2017

Section: Full talk for data engineering track Technical level: Advanced


Analysis of relationship between entities is at the heart of data mining problems. There are many metrics used for association mining like support, confidence, lift, mutual information etc. However many of these measures provide conflicting results about the interestingness of the association. Therefore it becomes very important to understand how to evaluate metrics for an application.


A fundamental step in pattern mining of transactional datasets is the extraction of frequent and interesting itemsets - a set of entities connected by the frequently occurring relationships between them. For instance, identifying from housing purchase data the correlations to age and income groups they belong to can lead to explainable relationships between the different data points which in turn leads to knowledge discovery. This kind of analysis is often confounded by the presence of spurious correlations and data sparsity especially in e-commerce where much of the traffic is often directed to a small percentage of the catalog.

Data mining is the process of discovering previously unknown interesting relationships which can be used to increase customer engagement and boost the sales. One very common and interesting example of this would be the discovery of strong link between diapers and beers from transactional data of walmart. This association was definitely not intuitive because you would expect customers to buy other baby products along with diapers. It was then suspected that these purchases were made by fathers baby sitting on weekends. Even Though this was found to be a strong correlation we must not forget that correlation doesn’t imply causation. A strong correlation is an interesting insight from user behaviour which can be fed back to increase customer interaction.

The goal of this talk is to set the stage for mining of associations, present commonly used techniques, followed by objective measures of interestingness of associations and finally, present an analysis on real customer datasets. In the case of sites attracting large amounts of customer traffic the main challenge is to make this process computationally effective.
We will pay special attention to low-traffic situations so that we can mine for interesting patterns without requiring large amount of data to support it. One way in which we achieve this is to build higher-level abstractions for entities such as the brand of the product rather than the product itself. In addition to this domain knowledge and transitive connections can be used to help with the cold-start problem.

Brief outline of the talk -

What is association mining

Motivation - Market basket analysis

Association mining: Basic concepts

  • Frequent ItemSet Generation

  • Association Generation

    • Apriori algorithm
    • FP growth algorithm

Evaluation of association patterns

  • Measures of interestingness

  • Case studies to compare interestingness measures

Challenges of mining sparse dataset

Unearthing unseen gems

  • Motivation for mining transitive or indirect association

  • Popular approaches

Key takeaways

Speaker bio

Simrat is a Data Scientist, Engineering Ninja and Inspector Gadget at Mad Street Den. She builds data platforms and models to make sense of user and product data in e-commerce online retail.




  • Abhishek Balaji (@booleanbalaji) Reviewer 2 years ago

    Hi Simrat,

    Please upload the draft slides so we can evaluate this talk.

    • Simrat Hanspal (@simrathanspal) Proposer 2 years ago (edited 2 years ago)

      Hi Abhishek, I have uploaded the draft slides.

  • Zainab Bawa (@zainabbawa) Reviewer 2 years ago

    How do you identify and eliminate biases when mining associations?

    • Simrat Hanspal (@simrathanspal) Proposer 2 years ago

      Hi Zainab, Can you please elaborate on what bias you are referring to.
      Technical ML bias causes the underfitting problem while the cognitive real life bias causes the overfitting problem.

      • Zainab Bawa (@zainabbawa) Reviewer 2 years ago

        My question took off from the diapering and beer example which has been critiqued for false association.

      • Zainab Bawa (@zainabbawa) Reviewer 2 years ago

        My question took off from the diapering and beer example which has been critiqued for false association.

  • Simrat Hanspal (@simrathanspal) Proposer a year ago

    Zainab, sorry for missing out your comment. Below is my response -

    Diaper-Beer is a popular example of interesting associations which are unintuitive and not obvious.
    Even though diaper and beer might not make sense together qualitatively if there is a strong correlation it might be considered interesting quantitatively. From retail perspective these can be used to improve sales and services for greater customer satisfaction.

    In the talk as well I would be delving into how we can mine such interesting associations.
    Hope this answers your question.

Login with Twitter or Google to leave a comment