Hi all. I hope this is the right place. It seems more like a Python problem=
 than an Ubuntu problem.

I use Python on Ubuntu 12.04. Recently I upgraded Python from 2.7.3 to 2.7.=
12. I did the upgrade via the source, i.e. download (from https://www.pytho=
n.org/downloads/release/python-2712/ ), tar, ./config, sudo make install.

After the upgrade, python could no longer see the 3rd party modules I use (=
numpy, scipy, matplotlib, cython). I can point PYTHONPATH to the cython fil=
es, but I can't work out how to include numpy, scipy and matplotlib.

It's no good asking Ubuntu to upgrade, because apt-get tells me that numpy,=
 scipy and matplotlib are all up to date.

So what do I do so I can see these packages again?

p.s. results of a few commands:

$ which python
/usr/local/bin/python

$ find /usr -name 'numpy*'
/usr/share/pyshared/numpy-1.6.1.egg-info
/usr/share/pyshared/scipy/weave/numpy_scalar_spec.py
/usr/share/pyshared/numpy
/usr/share/pyshared/numpy/core/include/numpy
/usr/share/pyshared/numpy/core/include/numpy/numpyconfig.h
/usr/share/pyshared/numpy/numarray/include/numpy
/usr/share/pyshared/numpy/testing/numpytest.py
/usr/share/pyshared/numpy/distutils/numpy_distribution.py
/usr/share/numpy
/usr/lib/python2.7/dist-packages/numpy-1.6.1.egg-info
/usr/lib/python2.7/dist-packages/scipy/weave/numpy_scalar_spec.py
/usr/lib/python2.7/dist-packages/scipy/weave/numpy_scalar_spec.pyc
/usr/lib/python2.7/dist-packages/numpy
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/numpyconfig.h
/usr/lib/python2.7/dist-packages/numpy/numarray/include/numpy
/usr/lib/python2.7/dist-packages/numpy/testing/numpytest.py
/usr/lib/python2.7/dist-packages/numpy/testing/numpytest.pyc
/usr/lib/python2.7/dist-packages/numpy/distutils/numpy_distribution.pyc
/usr/lib/python2.7/dist-packages/numpy/distutils/numpy_distribution.py
/usr/lib/pyshared/python2.7/numpy
/usr/local/lib/python2.7/dist-packages/Cython/Includes/numpy
/usr/include/python2.7/numpy
/usr/include/python2.7_d/numpy
/usr/include/numpy

.... and fairly similar results for scipy and matplotlib.

My bash PYTHONPATH is:

PYTHONPATH=3D:/home/peter/dvd/software/python:/home/peter/Cython-0.25.2

My sys.path in python is:

>>> sys.path
['', '/home/peter', '/home/peter/dvd/software/python', '/home/peter/Cython-=
0.25.2', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/l=
ocal/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/l=
ocal/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/usr/=
local/lib/python2.7/site-packages']

On Tue, Dec 20, 2016 at 4:08 PM,  <ozpeterballard@gmail.com> wrote:
> I use Python on Ubuntu 12.04. Recently I upgraded Python from 2.7.3 to 2.7.12. I did the upgrade via the source, i.e. download (from https://www.python.org/downloads/release/python-2712/ ), tar, ./config, sudo make install.
>
> After the upgrade, python could no longer see the 3rd party modules I use (numpy, scipy, matplotlib, cython). I can point PYTHONPATH to the cython files, but I can't work out how to include numpy, scipy and matplotlib.
>
> It's no good asking Ubuntu to upgrade, because apt-get tells me that numpy, scipy and matplotlib are all up to date.
>
> So what do I do so I can see these packages again?

First off, you're running a still-in-support old version of Ubuntu, so
check whether you can upgrade Python using apt-get. Check not just the
version number, but any features you need. I don't know whether Ubuntu
do this, but Red Hat often backport bug fixes and security patches
onto older builds of Python; what you get is named according to the
oldest component in it, but might well have a number of newer
features.

But once you decide to build Python from source, you have to choose
between overwriting your existing Python (probably in /usr/bin) or
creating a new one (in /usr/local/bin). By default, untarring,
configuring without args, building, and installing, will give you
something in /usr/local/bin - which most likely means you're leaving
behind all the Python modules installed with apt-get. You also won't
touch anything that uses libpython, or anything like that. This is the
safe option. So, how do you get numpy etc into your new Python? Most
likely, using pip. Try this:

sudo apt-get build-dep python-numpy
sudo python -m pip install numpy

That'll build numpy from source. (I may have the package name wrong;
that's what it's called on my Debian, but it might be different on
Ubuntu.) It'll take a while, so if you're on a system with a slow CPU
and/or not much memory, be patient. But that should get numpy going
for you.

ChrisA

Thanks Chris for replying, but it didn't work. The upgrade happened, but st=
ill python can't see numpy! So it seems to be a path problem. The numpy (an=
d scipy and matplotlib) files are there, so surely in principle it's a simp=
le matter of pointing my python path at them?

Any ideas how?

(I should add, the reason I upgraded python from source in the first place =
is that "apt-get" claimed python was up to date).

p.s. Here's an abridged log:

peter@sirboris:~$ which python
/usr/local/bin/python

peter@sirboris:~$ pip
The program 'pip' is currently not installed.  You can install it by typing=
:
sudo apt-get install python-pip

peter@sirboris:~$ sudo apt-get install python-pip
[lots of output, pip installed]

peter@sirboris:~$ which pip
/usr/bin/pip

peter@sirboris:~$ sudo pip install numpy
Requirement already satisfied (use --upgrade to upgrade): numpy in /usr/lib=
/python2.7/dist-packages
Cleaning up...

peter@sirboris:~$ sudo pip install numpy --upgrade
[lots and lots of output, successful install]

peter@sirboris:~$ python
Python 2.7.12 (default, Dec 11 2016, 22:16:38)=20
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named numpy
>>> import sys
>>> sys.path
['', '/home/peter', '/home/peter/dvd/software/python', '/home/peter/Cython-=
0.25.2', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/l=
ocal/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/l=
ocal/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/usr/=
local/lib/python2.7/site-packages']

# [Then I thought, maybe "pip" and "python -m pip" are different, so...]

peter@sirboris:~$ sudo python -m pip install numpy
/usr/local/bin/python: No module named pip

On Tue, Dec 20, 2016 at 5:19 PM,  <ozpeterballard@gmail.com> wrote:
> Thanks Chris for replying, but it didn't work. The upgrade happened, but =
still python can't see numpy! So it seems to be a path problem. The numpy (=
and scipy and matplotlib) files are there, so surely in principle it's a si=
mple matter of pointing my python path at them?
>
> Any ideas how?

You just installed pip into your /usr/bin/python, not
/usr/local/bin/python. You'll need to get pip in that Python and use
that. I'm not sure the best way to install pip into a 2.7 - someone
else on this list can advise.

ChrisA

Le 20/12/16 � 08:45, Chris Angelico a �crit :
> On Tue, Dec 20, 2016 at 5:19 PM,  <ozpeterballard@gmail.com> wrote:
>> Thanks Chris for replying, but it didn't work. The upgrade happened, but still python can't see numpy! So it seems to be a path problem. The numpy (and scipy and matplotlib) files are there, so surely in principle it's a simple matter of pointing my python path at them?
>>
>> Any ideas how?
> You just installed pip into your /usr/bin/python, not
> /usr/local/bin/python. You'll need to get pip in that Python and use
> that. I'm not sure the best way to install pip into a 2.7 - someone
> else on this list can advise.
>
> ChrisA

It seems you have shadowed your default python2 with the new one. A very 
bad idea.

You can have two python2 but you can't replace the default version.

The default is in /usr/lib and the new is in /usr/locale/lib

The third party libs are into the default version. For this reason 
there's probably some applications that are broken. (apport, bzr, 
synaptic?, ...)

The link /usr/bin/python *MUST* point to the 2.7.3 version, if not, 
restore it.

One solution for use the new version is to create the necessary symbolic 
links into /usr/locale/lib/python2.7/dist-packages which point to 
/usr/lib/python2.7/dist-packages numpy, scipy and others.

In fact, this is the same think when you create a virtual environment, 
the libs are not copied but linked.


VVV

On Tue, Dec 20, 2016 at 7:36 PM, Vincent Vande Vyvre
<vincent.vande.vyvre@telenet.be> wrote:
> It seems you have shadowed your default python2 with the new one. A very bad
> idea.

Only because /usr/local/bin is ahead of /usr/bin in PATH.

> The link /usr/bin/python *MUST* point to the 2.7.3 version, if not, restore
> it.

It does. When he installs pip via apt, it manages his original 2.7.3.
Anything that explicitly shebangs to /usr/bin/python will be
unaffected by this.

ChrisA

Le 20/12/16 � 10:13, Chris Angelico a �crit :
> On Tue, Dec 20, 2016 at 7:36 PM, Vincent Vande Vyvre
> <vincent.vande.vyvre@telenet.be> wrote:
>> It seems you have shadowed your default python2 with the new one. A very bad
>> idea.
> Only because /usr/local/bin is ahead of /usr/bin in PATH.
That's the problem.
>
>> The link /usr/bin/python *MUST* point to the 2.7.3 version, if not, restore
>> it.
> It does. When he installs pip via apt, it manages his original 2.7.3.
> Anything that explicitly shebangs to /usr/bin/python will be
> unaffected by this.
>
> ChrisA

Not sure, when the OP enter python into a terminal he have:

peter@sirboris:~$ python
Python 2.7.12 (default, Dec 11 2016, 22:16:38)

So, python point now to the new one. No? VVV

On Tue, Dec 20, 2016 at 8:28 PM, Vincent Vande Vyvre
<vincent.vande.vyvre@telenet.be> wrote:
>>> The link /usr/bin/python *MUST* point to the 2.7.3 version, if not,
>>> restore
>>> it.
>>
>> It does. When he installs pip via apt, it manages his original 2.7.3.
>> Anything that explicitly shebangs to /usr/bin/python will be
>> unaffected by this.
>>
>> ChrisA
>
>
> Not sure, when the OP enter python into a terminal he have:
>
> peter@sirboris:~$ python
> Python 2.7.12 (default, Dec 11 2016, 22:16:38)
>
> So, python point now to the new one. No? VVV

So when you type 'python' at the command line, it searches $PATH. But
you don't need to worry about system scripts; they'll use an explicit
"#!/usr/bin/python", which means they'll keep running 2.7.3.

(Unless, of course, the goal of upgrading was actually to change the
system Python, in which case you want to do things _very_
differently.)

ChrisA

OK thanks both of you. I didn't realise there was a distinction between system python and other (user/personal) python. Yes, python2.7.3 is still there in /usr/bin/python .

/usr/bin/python appears to be getting numpy and scipy from /usr/lib/python2.7/dist-packages . So I added that to PYTHONPATH and reran python (i.e. /usr/local/bin/python, which is 2.7.12). 

But now when I "import numpy" I get an error message which ends in:
 File "/usr/lib/python2.7/dist-packages/numpy/core/__init__.py", line 5, in <module>
    import multiarray
ImportError: /usr/lib/python2.7/dist-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS4_AsUnicodeEscapeString

So for now I give up.

But the good news is that I have learned that my "old" python (2.7.3) is still on my machine. So for now I am renaming /usr/local/bin/python to /usr/local/bin/python2.7.12 , and going back to using python 2.7.3.