[Python] Strange $PATH related issue when running Ansible

I have 2 RHEL 8.10 machines. Both completely patched, and the same version of Ansible is installed on both:

Server A:
dnf list installed | grep ansible
ansible-core.x86_64 2.16.3-2.el8

Server B:
dnf list installed | grep ansible
ansible-core.x86_64 2.16.3-2.el8


Also, both have the same versions of Python3 installed, python 3.6.8 and python 3.12.8.

When I run

ansible --version


on both servers, I get a different result:

Server A:
ansible --version
ansible [core 2.11.12]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/sysadmin/.ansible/plugins/modules',
'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
ansible collection location =
/home/sysadmin/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.6.8 (default, Nov 15 2024, 08:11:39) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)]
jinja version = 3.0.3
libyaml = True

Server B:
ansible [core 2.16.3]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/sysadmin/.ansible/plugins/modules',
'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.12/site-packages/ansible
ansible collection location =
/home/sysadmin/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible
python version = 3.12.8 (main, Dec 9 2024, 15:25:01) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] (/usr/bin/python3.12)
jinja version = 3.1.2
libyaml = True


The configuration on Server B is how I want it to be. As you can see, the ansible executable location is different:

Server A:
executable location = /usr/local/bin/ansible

Server B:
executable location = /usr/bin/ansible


I have no clue why this is. As far as I can remember, both machines were configured in the same way.

When I look at the paths this user has when running ansible, I see this :

Server A:
echo $PATH
/home/sysadmin/.local/bin:/home/sysadmin/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/sysadmin/.local/bin:/home/sysadmin/bin

Server B:
echo $PATH
/home/sysadmin/.local/bin:/home/sysadmin/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin


Also different, but on both servers the path /usr/local/bin comes before /usr/bin.

I tried renaming the ansible executable on server A from /usr/local/bin/ansible to /usr/local/bin/ansible_old, but now I get:

-bash: /usr/local/bin/ansible: No such file or directory


So I'm guessing the (wrong) location is cached somewhere as it still wants to find the executable in /usr/local/bin. Since it isn't there anymore, I would have expected that the $PATH variable would have been traversed further, and ansible in /usr/bin would have been found.

There are no aliases pointing ansible to /usr/local/bin/ansible...

Where do I go wrong ?

Continue reading...