[Python] Still Having Trouble Integrating LinearFold with Arnie's bpps.py – KeyError...

I am modifying bpps.py in the Arnie Python package to integrate LinearFold for RNA secondary structure predictions. However, I keep encountering "KeyError: 'linearfold_v'", and issues when running bpps.py, even though all dependencies appear to be installed correctly.

To debug, I am working with the following files:

1. bpps.py – Contains the main logic for base-pairing probability calculations. This file handles package selection and calls external RNA secondary structure prediction tools.
2. arnie_config.txt – Registers paths to RNA folding executables (e.g., ViennaRNA, LinearFold, etc.). This is where I define locations for LinearFold binaries.
3. package_options.py – Defines available packages and how they are processed in bpps.py. It should contain mappings for LinearFold and LinearPartition.

Despite ensuring that all required dependencies are installed and accessible, I still get KeyError: 'linearfold_v' when calling bpps(seq, package="linearfold_v").

1. Ensured LinearFold is Installed and Accessible


My linearfold_v binary is located at:

/home/vboxuser/LinearFold/bin/linearfold_v


Similarly, linearfold_c, linearfold_e, and linearpartition_* are in their respective locations.

To interface with LinearFold, I added the following function in bpps.py (the entire code is included at the bottom:

def bpps_linearfold(sequence):
"""Run LinearFold on a sequence and return base-pairing probabilities."""
import subprocess

linearfold_path = "/home/vboxuser/LinearFold/bin/linearfold_v"
command = [linearfold_path, "--prob", sequence]

process = subprocess.run(command, text=True, capture_output=True)
if process.returncode != 0:
raise RuntimeError(f"LinearFold failed: {process.stderr}")

return process.stdout


This function runs without errors when executed independently.

2. Updated bpps.py to Recognize LinearFold


To integrate LinearFold, I updated bpps.py to process linearfold_v correctly:

if linear:
print(f"DEBUG: Linear is set to True, checking package: {package}")

if package in ["linearfold_v", "linearfold_c", "linearfold_e"]:
print(f"DEBUG: Calling bpps_linearfold with {package}")
return bpps_linearfold(sequence)

elif package in ["linearpartition_v", "linearpartition_c", "linearpartition_e"]:
print(f"DEBUG: Calling bpps_linearpartition with {package}")
return bpps_linearpartition(sequence, tmp_file)

else:
print(f"DEBUG: Unexpected package {package} for linear processing!")


However, calling:

# test_bpps.py
from arnie.bpps import bpps

# Test sequence
seq = "GGCGGAUGCAUGGAGCUAAUGGCUGUAGCUCA"

# Attempt to call LinearFold
result = bpps(seq, package="linearfold_v")

print(result)
)


Still raises:

KeyError: 'linearfold_v'


indicating linearfold_v is not being recognized in Arnie's package system.

3. Checked package_options.py for Proper Inclusion


In package_options.py, I verified that linearfold_v is listed:

package_options = {
'vienna': {'package': 'vienna'},
'vienna_2': {'package': 'vienna_2'},
'contrafold_1': {'package': 'contrafold_1'},

'linearfold_v': {'package': 'linearfold', 'type': 'v'},
'linearfold_c': {'package': 'linearfold', 'type': 'c'},
'linearfold_e': {'package': 'linearfold', 'type': 'e'},

'linearpartition_v': {'package': 'linearpartition', 'type': 'v'},
'linearpartition_c': {'package': 'linearpartition', 'type': 'c'},
'linearpartition_e': {'package': 'linearpartition', 'type': 'e'}
}


This suggests that linearfold_v is defined, but not being properly registered within Arnie.

4. Verified arnie_config.txt Contains LinearFold Paths


I checked my arnie_config.txt file, located at:

~/.arnie_config.txt


and ensured it includes:

vienna: /home/vboxuser/miniconda3/envs/rna_env/bin/RNAfold
linearfold: /home/vboxuser/LinearFold/bin/linearfold
linearfold_v: /home/vboxuser/LinearFold/bin/linearfold_v
linearfold_c: /home/vboxuser/LinearFold/bin/linearfold_c
linearfold_e: /home/vboxuser/LinearFold/bin/linearfold_e
linearpartition: /home/vboxuser/LinearPartition/bin/linearpartition
eternafold: /home/vboxuser/miniconda3/envs/rna_env/bin/eternafold
TMP: ./tmp


To verify if Arnie recognizes LinearFold, I ran:

python -c "import arnie.utils; print(arnie.utils.load_package_locations())"


Output:

• linearfold_v does not appear, meaning Arnie is not detecting it.

Questions:

1. Is there another step needed to register new packages in Arnie?
2. How can I force Arnie to recognize LinearFold from arnie_config.txt?

5. Debugging Statements Confirm Package Is Not Found


After adding debugging print statements to bpps.py:

DEBUG: Linear is set to True, checking package: linearfold_v
DEBUG: Unexpected package linearfold_v for linear processing!


This confirms that Arnie is not detecting linearfold_v as a valid package.

6. Current Error Output


Running:

from arnie.bpps import bpps
seq = "GGCGGAUGCAUGGAGCUAAUGGCUGUAGCUCA"
result = bpps(seq, package="linearfold_v")


Results in:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/vboxuser/miniconda3/envs/rna_env/lib/python3.9/site-packages/arnie/bpps.py", line 67, in bpps
else: _, tmp_file = pfunc(sequence, package=package, bpps=True, linear=linear, ...)
KeyError: 'linearfold_v'

What I Need Help With

1. Why is bpps.py not recognizing "linearfold_v", even though it’s in package_options.py?
2. How do I correctly add LinearFold to arnie_config.txt so Arnie can detect it?
3. Are there additional files or methods in Arnie that I need to modify to register a new package?
4. What other debugging steps should I try to ensure LinearFold is properly recognized?

Any help would be greatly appreciated!

arnie/bpps.py code

import os, re, sys
import subprocess as sp
import random, string
import numpy as np
from .utils import *
from .pfunc import pfunc

# load package locations from yaml file, watch! global dict
package_locs = load_package_locations()

def bpps(sequence, package='vienna', constraint=None, pseudo=False,
T=37, coaxial=True, linear=False, dna=False,
motif=None, dangles=True,param_file=None,reweight=None, beam_size=100, DEBUG=False, threshknot=False,
probing_signal=None, probing_kws=None,DIRLOC=None):

''' Compute base pairing probability matrix for RNA sequence.

Args:
sequence (str): nucleic acid sequence
T (float): temperature (Celsius)
linear (bool): call LinearPartition to estimate Z in Vienna or Contrafold
constraint (str): structure constraint (functional in vienna, contrafold, rnastructure)
motif (str): argument to vienna motif
pseudo (bool): (NUPACK only) include pseudoknot calculation
dangles (bool): dangles or not, specifiable for vienna, nupack
dna (bool): (NUPACK only) use SantaLucia 1998 parameters for DNA
coaxial (bool): coaxial stacking or not, specifiable for rnastructure, vfold
noncanonical(bool): include noncanonical pairs or not (for contrafold, RNAstructure (Cyclefold))
beam size (int): Beam size for LinearPartition base pair calculation.
DEBUG (bool): Output command-line calls to packages.
threshknot (bool): calls threshknot to predict pseudoknots (for contrafold with LinearPartition)

Possible packages: 'vienna_2', 'vienna_1','contrafold_1','contrafold_2',
'nupack_95','nupack_99','rnasoft_2007','rnasoft_1999','rnastructure','vfold_0','vfold_1'

Returns
array: NxN matrix of base pair probabilities
'''
package = package.lower()
try:
pkg, version = package.split('_')
except:
pkg, version = package, None

if motif is not None and pkg != 'vienna':
raise ValueError('motif option can only be used with Vienna.')

if pseudo and pkg != 'nupack':
raise ValueError('pseudoknot option only implemented with Nupack.')

if not dangles and pkg not in ['vienna','nupack']:
print('Warning: %s does not support dangles options' % pkg)
if not coaxial and pkg not in ['rnastructure','vfold']:
print('Warning: %s does not support coaxial options' % pkg)
if linear and pkg not in ['vienna','contrafold','eternafold']:
print('Warning: LinearPartition only implemented for vienna, contrafold, eternafold.')

if pkg=='nupack':
return bpps_nupack_(sequence, version = version, dangles = dangles, T = T, pseudo=pseudo, dna=dna)

elif pkg=='vfold':
return bpps_vfold_(sequence, version = version, T = T, coaxial = coaxial)
elif package == "MyGreatPackage":
return bpps_myGreatPackage_(sequence, myNewArgument=myDefaultValue)

else:

_, tmp_file = pfunc(sequence, package=package, bpps=True, linear=linear,
motif=motif, constraint=constraint, T=T, coaxial=coaxial, probing_signal=probing_signal, probing_kws=probing_kws, DIRLOC=package_locs[package],
dangles=dangles, param_file=param_file,reweight=reweight, beam_size=beam_size, DEBUG=DEBUG, threshknot=threshknot)

if linear:
print(f"DEBUG: Linear is set to True, checking package: {package}")
if package in ["linearfold_v", "linearfold_c", "linearfold_e"]:
print(f"DEBUGL Calling bpps_linearfold with {package}")
return bpps_linearfold(sequence, tmp_file)
elif package in ["linearpartition_v", "linearpartition_c", "linearpartition_e"]
print(f"DEBUGL Calling bpps_linearpartition with {package}")
return bpps_linearpartition_(sequence, tmp_file)
else:

if 'contrafold' in pkg:
return bpps_contrafold_(sequence, tmp_file)
if package=='eternafold':
return bpps_contrafold_(sequence, tmp_file)
elif 'vienna' in pkg:
return bpps_vienna_(sequence, tmp_file)
elif 'rnasoft' in pkg:
return bpps_rnasoft_(sequence, tmp_file)
elif 'rnastructure' in pkg:
return bpps_rnastructure_(sequence, tmp_file, coaxial=coaxial)

else:
raise RuntimeError('package not yet implemented')

def bpps_vienna_(sequence, tmp_file):

dot_fname = tmp_file

probs=np.zeros([len(sequence), len(sequence)])
with open(dot_fname,'r') as f:
for line in f.readlines():
if 'ubox' in line:
try:
i, j, p, _ = line.split()
i, j, p = int(i)-1, int(j)-1, float(p)**2
probs[i,j] = p
probs[j,i] = p
except:
pass
os.remove(dot_fname)
return probs

def bpps_contrafold_(sequence, tmp_file):

fname = tmp_file

probs=np.zeros([len(sequence), len(sequence)])

for line in open(fname).readlines():
if len(line.split(':')) > 1:
first_ind = int(line.split()[0])-1
for x in line.split()[2:]:
second_ind = int(x.split(':')[0])-1
p = float(x.split(':')[1])
probs[first_ind, second_ind] = p
probs[second_ind, first_ind] = p

os.remove(fname)

return probs

def bpps_rnasoft_(sequence, tmp_file):
fname = tmp_file

probs=np.zeros([len(sequence), len(sequence)])
for line in open(fname).readlines():
i,j,p = int(line.split()[0]), int(line.split()[1]), float(line.split()[2])
probs[i,j] = p
probs[j,i] = p

os.remove(fname)

return probs

def bpps_nupack_(sequence, version='95', T=37, dangles=True, pseudo=False,dna=False):

if not version: version='95'

nupack_materials={'95': 'rna1995', '99': 'rna1999'}

if dna:
material='dna1998'
else:
material=nupack_materials[version]

DIR = package_locs['nupack']

if dangles:
dangle_option='some'
else:
dangle_option='none'

seqfile = write([sequence])

command=['%s/pairs' % DIR, '%s' % seqfile.replace('.in',''),
'-T', str(T), '-material', material, '-dangles', dangle_option, '-cutoff', '0.0000000001']

if pseudo:
command.append('--pseudo')
p = sp.Popen(command, stdout=sp.PIPE, stderr=sp.PIPE)

stdout, stderr = p.communicate()

if p.returncode:
raise Exception('Nupack pfunc failed: on %s\n%s' % (sequence, stderr))

ppairs_file = '%s.ppairs' % seqfile.replace('.in','')
os.remove(seqfile)

probs=np.zeros([len(sequence), len(sequence)])

with open(ppairs_file, 'r') as f:
for line in f.readlines():
if not line.startswith('%'):
fields = line.split()
if len(fields) > 1:
if int(fields[1]) <= len(sequence):
i, j, p = int(fields[0])-1, int(fields[1])-1, float(fields[2])
probs[i,j] = p
probs[j,i] = p
os.remove(ppairs_file)

return probs

def bpps_rnastructure_(sequence, tmp_file, coaxial=True, DEBUG=False):

DIR = package_locs['rnastructure']

pfsfile = tmp_file #'%s/rnastructtmp.pfs' % package_locs['TMP']
outfile = '%s.probs' % (tmp_file.replace('.pfs',''))
command = ['%s/ProbabilityPlot' % DIR, pfsfile, outfile, '-t', '-min', '0.0000000001']

probs=np.zeros([len(sequence), len(sequence)])

if DEBUG: print(' '.join(command))
p = sp.Popen(command, stdout=sp.PIPE, stderr=sp.PIPE)

stdout, stderr = p.communicate()

if DEBUG:
print('stdout')
print(stdout)
print('stderr')
print(stderr)

if p.returncode:
raise Exception('RNAstructure ProbabilityPlot failed: on %s\n%s' % (seq, stderr))

with open(outfile, 'r') as f:
for line in f.readlines()[2:]:
fields = line.split()
i, j, p = int(fields[0])-1, int(fields[1])-1, 10**(-1*float(fields[2]))
probs[i,j] = p
probs[j,i] = p

os.remove(outfile)
os.remove(pfsfile)
return probs

def bpps_vfold_(sequence, version='0',T=37, coaxial=True, DEBUG=False):
#available versions: 0 for Turner 04 params, 1 for Mfold 2.3 params

DIR = package_locs["vfold"]

cwd = os.getcwd()
os.chdir(DIR) #vfold precompiled binaries don't work being called from elsewhere

if DEBUG: print(os.getcwd())

seqfile = write([sequence])

outfile = filename()+'.pij'

if sys.platform=="linux":
platform='linux'
elif sys.platform=="darwin":
platform='mac'
elif sys.platform=="win32":
platform='win'
else:
raise RuntimeError('Vfold has binaries for linux, macOS, and win')

command = ['./Vfold2d_npk_%s.o %d %d %s %s %d' % (platform, int(coaxial), T, seqfile, outfile, int(version))]

if DEBUG: print(' '.join(command))

p = sp.Popen(command, stdout=sp.PIPE, stderr=sp.PIPE, shell=True)

stdout, stderr = p.communicate()
os.chdir(cwd)

if DEBUG:
print('stdout')
print(stdout)
print('stderr')
print(stderr)
if p.returncode:
raise Exception('Vfold2d_npk failed: on %s\n%s' % (sequence, stderr))

os.remove(seqfile)
probs = np.zeros([len(sequence),len(sequence)])
p_ij_output = np.loadtxt(outfile,usecols=(0,2,3)) #col 0: set of inds 1, col 1: set of inds 2, col 2: bpp

for i,j,p in p_ij_output:
probs[int(i-1),int(j-1)] = p
probs[int(j-1),int(i-1)] = p
os.remove(outfile)

return probs
#output: take second field of last line for Z


def bpps_linearpartition_(sequence, tmp_file):

fname = tmp_file

probs=np.zeros([len(sequence), len(sequence)])

for line in open(fname,'r').readlines():
if len(line.strip())>0:
first_ind, second_ind, p = line.strip().split(' ')
first_ind = int(first_ind)-1
second_ind = int(second_ind)-1
p = float(p)
probs[first_ind, second_ind] = p
probs[second_ind, first_ind] = p

os.remove(fname)

return probs

def bpps_myGreatPackage_(sequence, myNewArgument=True):
return bp_array

def bpps_linearfold(sequence):
# Run LinearFold on a sequence and return base-pairing probabilities
import subprocess

linearfold_path = "/home/vboxuser/LinearFold/bin/linearfold_v" #Will change when running linearfold_c or e
command = [linearfold_path, "--prob", sequence]

process = subprocess.run(command, text=True, capture_output=True)

if process.returncode !=0:
raise RuntimeError(f"Linearfold failed: {process.stderr}")
return process.stdout

Continue reading...