I am not sure title is well describes the problem but I will explain it step by step. I have a correlation matrix of genes (10k x 10k) I convert this correlation matrix to pairwise dataframe (upper triangle) (around 100m row) Then I have gold-standard TERMS table around 5k rows and columns are ID and used_genes What I do: I iterate of each Gold-standard complex row. Convert used_gene list to pairwise, like geneA-geneB, geneA-geneC etc. Check those complex-row gene pairs in the stacked correlation pairs. If they are exist I put column TP=1, if not TP=0. Based on the TP counts I calculate precision, recall, and area under the curve score. I will share my function below, with 100m stacked df, and 5k terms It takes around 25 minutes, and I am trying to find a way to reduce the time. def compute_per_complex_pr(corr_matrix, terms_df): pairwise_df = binary(corr_matrix) pairwise_df = quick_sort(pairwise_df).reset_index(drop=True) # Precompute a mapping from each gene to the row indices in the pairwise DataFrame where it appears. gene_to_pair_indices = {} for i, (gene_a, gene_b) in enumerate(zip(pairwise_df["gene1"], pairwise_df["gene2"])): gene_to_pair_indices.setdefault(gene_a, []).append(i) gene_to_pair_indices.setdefault(gene_b, []).append(i) # Initialize AUC scores (one for each complex) with NaNs. auc_scores = np.full(len(terms_df), np.nan) # Loop over each gene complex for idx, row in terms_df.iterrows(): gene_set = set(row.used_genes) if len(gene_set) < min_complex_size: # Skip small complexes. continue # Collect all row indices in the pairwise data where either gene belongs to the complex. candidate_indices = set() for gene in gene_set: candidate_indices.update(gene_to_pair_indices.get(gene, [])) candidate_indices = sorted(candidate_indices) if not candidate_indices: continue # Select only the relevant pairwise comparisons. sub_df = pairwise_df.loc[candidate_indices] # A prediction is 1 if both genes in the pair are in the complex; otherwise 0. predictions = (sub_df["gene1"].isin(gene_set) & sub_df["gene2"].isin(gene_set)).astype(int) if predictions.sum() == 0: continue # Compute cumulative true positives and derive precision and recall. true_positive_cumsum = predictions.cumsum() precision = true_positive_cumsum / (np.arange(len(predictions)) + 1) recall = true_positive_cumsum / true_positive_cumsum.iloc[-1] if len(recall) < 2 or recall.iloc[-1] == 0: continue auc_scores[idx] = metrics.auc(recall, precision) # Add the computed AUC scores to the terms DataFrame. terms_df["auc_score"] = auc_scores return terms_df Continue reading...
