How to install a newer version of Python on EPIC or RIO

Update! Be sure and read the post after this one!

We have had a few customers want to do ‘more’ with Python on EPIC and RIO.
The problem they quickly come across is that the version on each is a little dated for their needs.
They specifically need to install either Pandas or a newer version of Numpy.

The solution is to install a newer version of Python alongside the existing version. Note that you can’t just update the version already on the EPIC - it must remain the same as it is used by many other packages on the EPIC/RIO and updating it will break those packages. (Hint, you will need a paperclip to recover).

The key is the ‘altinstall’.
Here are the steps to install Pandas which has Numpy as a prerequisite.

First, update your repo and then install some dependencies and required libraries.

sudo apt-get update
sudo apt-get install libz-dev
sudo apt-get install libffi-dev
sudo apt-get install openssl-dev
sudo apt-get install libreadline-dev
sudo apt-get install libpcap-dev

Get a newer version of python

wget -O ~/python3.9.7.tgz https://www.python.org/ftp/python/3.9.7/Python-3.9.7.tgz

Untar it.

tar -zxvf python3.9.7.tgz

cd into the directory so we can configure and make it. Note the make process will take around 20 minutes.

cd Python-3.9.7/
./configure
sudo make -j4 altinstall

Note that altinstall is critical - you must not overwrite the version of Python that is on the EPIC/RIO.
BTW, the -j4 uses all 4 cores on an EPIC, if you are doing this on RIO, drop the -j4 from the command.

Now install pandas- note this will take about 2 hours to get all the wheel packages, also this will install numpy 1.21.2 which is newer than the version in the Opto repo (1.7.0).

Also note you must explicitly use the pip3.9 installer, not the default EPIC pip installer.

sudo pip3.9 install pandas

Please note that this whole process will take around 3 hours and will sometimes look like nothing is happening.
Just be patient and wait for each step to finish.
@torchard and I have tested it end to end successfully many times.

Lastly, note that from here on out, you will need to call out ‘python3.9’ for any commands as well.

If you have read this far you probably know it goes with out saying, all this is like any other fun-and-games with shell, we don’t support it and if you do a firmware update, you will need to redo all these steps plus any other changes you have made in shell.

2 Likes

As new versions of Linux packages are release, the dependencies of those packages change, this can cause build processes to break. (Which is out side of Opto’s control).

On top of that when we release new groov firmware versions, we make changes to our package repo. This is in our control and we generally only add packages that we know work Ok with our Yocto build of Linux.

Bottom line of those two variables is sometimes the packages don’t align any more and so something that worked months ago, no longer works.

Ok, so that’s the why, how do we install pandas and numpy with a new version of Python in Jan 2022?

The first few steps remain the same…

sudo apt-get update
sudo apt-get install libz-dev
sudo apt-get install libffi-dev
sudo apt-get install openssl-dev
sudo apt-get install libreadline-dev
sudo apt-get install libpcap-dev

Get a newer version of python

wget -O ~/python3.9.7.tgz https://www.python.org/ftp/python/3.9.7/Python-3.9.7.tgz

Untar it.

tar -zxvf python3.9.7.tgz

cd into the directory so we can configure and make it. Note the make process will take around 20 minutes.

cd Python-3.9.7/
./configure
su make -j4 altinstall

Now we need to make a small change, we have to call out the specific versions of the two main packages we are interested in.

sudo pip3.9 install numpy==1.21.2 pandas==1.3.5

After the build (which still takes a few hours due to all the packages that are built) you can check your built and installed versions:

pip3.9 list

The result of this command on my groov is as follows:

Package         Version
--------------- -------
numpy           1.21.2
pandas          1.3.5
pip             21.2.3
python-dateutil 2.8.2
pytz            2021.3
setuptools      57.4.0
six             1.16.

Going forward, @torchard and I are going to test this update/build process of part of our pre-flight checks for groov firmware updates.

1 Like

Hi Beno,

I want to install scikit-learn in EPIC and the EPIC shut down then I need to restart it.
Do you have any idea for it ?
Thanks.

Followed is the record -
bstech@opto-04-dc-e9:~$ sudo pip3.9 install -U scikit-learn
Password:
Collecting scikit-learn

Downloading scikit-learn-1.0.2.tar.gz (6.7 MB)
|████████████████████████████████| 6.7 MB 1.9 MB/s
Installing build dependencies … |Connection to 192.168.10.109 closed by remote host.
Connection to 192.168.10.109 closed.

I have spent many hours working on this trying to track down what the issue is.
As best as I can see so far, the issue is scipy. Or more accurately, one or two of the possibly three libraries required by scipy, namely BLAS, LAPACK or MKL. (For sure most likely one of the first two).
ATLAS and OpenBLAS are not trivial themselves to build and this is where the problem you see (and yes, I can reproduce it here) arises.

As an aside I do NOT recommend the -U option in pip. As this thread points out, the exact versions of the different packages we need to install are very carefully selected, when you throw a -U in there and do an update on all of them, a lot will be guaranteed to break.

I will continue to work on the libraries and report back. This could take some time to resolve.

1 Like

Hi Beno,

Thanks for your help and suggestion.
I’ll also keep do more testing and see any solution for it.