I have a binary 2D numpy array that represents transactions and items. For item I (represented as columns), I want to calculate the total number of transactions (represented as rows) that purchased this item, all items that were purchased alongside item i, and all transactions associated with those items. This is done to calculate support, confidence, and lift between items. My current implementation uses a nested loop but it takes too long for a large dataset. Is there a way to scan a matrix optimally when I need to scan the entirety of it each time? Here is my code with an example array. import numpy as np results = [] data = np.array([[0,1,0,1],[1,1,0,1],[0,0,0,1],[1,1,1,1]]) N = data.shape[0] #total transactions for i in range(4): # for each item i #transactions% where item i was purchased mask_i = data[:,i] == 1 support_i = data[mask_i].shape[0]/N for j in range(4): # check all items j #transactions where both i and j are purchased mask_j = data[:,j] == 1 support_j = data[mask_j].shape[0]/N #transactions% where j is purchased mask_c = np.logical_and(mask_i,mask_j) # mask for where both i and j were purchased confidence = (data[mask_c].shape[0]/N) / support_i lift = confidence/support_j results.append([i,j,confidence,lift]) #store combination The results list will show each identified combination of item with other items. [[0, 0, 1.0, 2.0], [0, 1, 1.0, 1.3333333333333333], [0, 2, 0.5, 2.0], [0, 3, 1.0, 1.0], [1, 0, 0.6666666666666666, 1.3333333333333333], [1, 1, 1.0, 1.3333333333333333], [1, 2, 0.3333333333333333, 1.3333333333333333], [1, 3, 1.0, 1.0], [2, 0, 1.0, 2.0], [2, 1, 1.0, 1.3333333333333333], [2, 2, 1.0, 4.0], [2, 3, 1.0, 1.0], [3, 0, 0.5, 1.0], [3, 1, 0.75, 1.0], [3, 2, 0.25, 1.0], [3, 3, 1.0, 1.0]] Continue reading...
