f



'reload M' doesn't update 'from M inport *'

I develop in an IDLE window.

Module M says 'from service import *'.
Next I correct a mistake in function 'service.f'.
Now 'service.f' works fine.

I do 'reload (service); reload (M)'.
The function 'M.f' still misbehaves.

'print inspect.getsource (service.f)' and
'print inspect.getsource (M.f)' shows the same 
corrected code. 

'print service.f' and 'print M.f' show different ids.

So I do 'del M; reload (M)'. Nothing changes.

I delete M again and run gc.collect () to really 
clean house. I reload M again and still nothing changes.
The id of the reloaded function 'M.f' is still the 
same as it was before the purge and so M.f still isn't 
fixed.  

I know I have more radical options, such as starting 
a new IDLE window. That would save me time, but 
I'd like to take the opportunity to understand what
is happening. Surely someone out there knows.

Frederic
 




0
Frederic
7/9/2010 1:02:25 PM
comp.lang.python 77058 articles. 6 followers. Post Follow

5 Replies
778 Views

Similar Articles

[PageSpeed] 35

In article <mailman.465.1278680555.1673.python-list@python.org>,
Frederic Rentsch  <anthra.norell@bluewin.ch> wrote:
>
>Module M says 'from service import *'.
>Next I correct a mistake in function 'service.f'.
>Now 'service.f' works fine.
>
>I do 'reload (service); reload (M)'.
>The function 'M.f' still misbehaves.

Absolutely!

>'print inspect.getsource (service.f)' and
>'print inspect.getsource (M.f)' shows the same 
>corrected code. 
>
>'print service.f' and 'print M.f' show different ids.
>
>So I do 'del M; reload (M)'. Nothing changes.
>
>I delete M again and run gc.collect () to really clean house. I reload
>M again and still nothing changes.  The id of the reloaded function
>'M.f' is still the same as it was before the purge and so M.f still
>isn't fixed.
>
>I know I have more radical options, such as starting a new IDLE
>window. That would save me time, but I'd like to take the opportunity
>to understand what is happening. Surely someone out there knows.

Take a look at sys.modules to get a better idea of what's happening.
(Maybe someone else will have time to write a longer answer.)

But really, relying on reload() is foolish in the general case because
it's nearly impossible to track down every single reference.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

"....Normal is what cuts off your sixth finger and your tail..."  --Siobhan
0
aahz
7/9/2010 3:47:51 PM
On Fri, 09 Jul 2010 15:02:25 +0200, Frederic Rentsch wrote:

> I develop in an IDLE window.
> 
> Module M says 'from service import *'. Next I correct a mistake in
> function 'service.f'. Now 'service.f' works fine.

from service import *

should be considered advanced functionality that is discouraged unless 
you really know what you are doing, precisely for the problems you are 
experiencing. You should try to avoid it.

But putting that aside, if you have done "from service import *" in 
module m, where are you getting "service.f" from? The only way that is 
possible is if you ALSO say "import service".


> I do 'reload (service); reload (M)'.
> The function 'M.f' still misbehaves.
> 
> 'print inspect.getsource (service.f)' and 'print inspect.getsource
> (M.f)' shows the same corrected code.

inspect.getsource always looks at the source code on disk, no matter what 
the byte code in memory actually says.

> 'print service.f' and 'print M.f' show different ids.
> 
> So I do 'del M; reload (M)'. Nothing changes.
> 
> I delete M again and run gc.collect () to really clean house. I reload M
> again and still nothing changes. The id of the reloaded function 'M.f'
> is still the same as it was before the purge and so M.f still isn't
> fixed.
>
> I know I have more radical options, such as starting a new IDLE window.
> That would save me time, but I'd like to take the opportunity to
> understand what is happening. Surely someone out there knows.

Yes. You have to understand importing. Let's start with the simple:

import m

In *very* simplified pseudo-code, this does:

look for module m in the global cache
if not there, then:
    search for m.py
    compile it to a Module object
    put the Module object in the cache
create a new name "m" in the local namespace
set the name "m" to the Module object in the cache

Now let's compare it to:

from m import f

look for module m in the global cache
if not there, then:
    search for m.py
    compile it to a Module object
    put the Module object in the cache
look for object named "f" in the Module object
create a new name "f" in the local namespace
set the name "f" to cached object

The important thing to notice is the the name "f" is a local variable. It 
doesn't, and can't, remember that it comes from module m. Reloading m 
can't do anything to f, because the connection is lost.

Now consider that the object "f" that came from m was itself imported 
from another module, "service". Reloading service doesn't help, because 
m.f doesn't know it came from service. Reloading m doesn't help, because 
all that does is run "from service import f" again, and that just fetches 
f from the global cache.

The simplest, easiest way of dealing with this is not to have to deal 
with it: don't use "from service import f", and ESPECIALLY don't use 
"from service import *". Always use fully-qualified importing:

import service
service.f

Now "reload(service)" should do what you expect.

The other way is not to bother with reload. It's not very powerful, only 
good for the simplest use in the interactive interpreter. Just exit the 
interpreter and restart it.


-- 
Steven
0
Steven
7/9/2010 3:58:38 PM
On Fri, 2010-07-09 at 15:58 +0000, Steven D'Aprano wrote:
> On Fri, 09 Jul 2010 15:02:25 +0200, Frederic Rentsch wrote:
> 
> > I develop in an IDLE window.
> > 
> > Module M says 'from service import *'. Next I correct a mistake in
> > function 'service.f'. Now 'service.f' works fine.
> 
> from service import *
> 
> should be considered advanced functionality that is discouraged unless 
> you really know what you are doing, precisely for the problems you are 
> experiencing. You should try to avoid it.
> 
> But putting that aside, if you have done "from service import *" in 
> module m, where are you getting "service.f" from? The only way that is 
> possible is if you ALSO say "import service".
> 
> 
> > I do 'reload (service); reload (M)'.
> > The function 'M.f' still misbehaves.
> > 
> > 'print inspect.getsource (service.f)' and 'print inspect.getsource
> > (M.f)' shows the same corrected code.
> 
> inspect.getsource always looks at the source code on disk, no matter what 
> the byte code in memory actually says.
> 
> > 'print service.f' and 'print M.f' show different ids.
> > 
> > So I do 'del M; reload (M)'. Nothing changes.
> > 
> > I delete M again and run gc.collect () to really clean house. I reload M
> > again and still nothing changes. The id of the reloaded function 'M.f'
> > is still the same as it was before the purge and so M.f still isn't
> > fixed.
> >
> > I know I have more radical options, such as starting a new IDLE window.
> > That would save me time, but I'd like to take the opportunity to
> > understand what is happening. Surely someone out there knows.
> 
> Yes. You have to understand importing. Let's start with the simple:
> 
> import m
> 
> In *very* simplified pseudo-code, this does:
> 
> look for module m in the global cache
> if not there, then:
>     search for m.py
>     compile it to a Module object
>     put the Module object in the cache
> create a new name "m" in the local namespace
> set the name "m" to the Module object in the cache
> 
> Now let's compare it to:
> 
> from m import f
> 
> look for module m in the global cache
> if not there, then:
>     search for m.py
>     compile it to a Module object
>     put the Module object in the cache
> look for object named "f" in the Module object
> create a new name "f" in the local namespace
> set the name "f" to cached object
> 
> The important thing to notice is the the name "f" is a local variable. It 
> doesn't, and can't, remember that it comes from module m. Reloading m 
> can't do anything to f, because the connection is lost.
> 
> Now consider that the object "f" that came from m was itself imported 
> from another module, "service". Reloading service doesn't help, because 
> m.f doesn't know it came from service. Reloading m doesn't help, because 
> all that does is run "from service import f" again, and that just fetches 
> f from the global cache.
> 
> The simplest, easiest way of dealing with this is not to have to deal 
> with it: don't use "from service import f", and ESPECIALLY don't use 
> "from service import *". Always use fully-qualified importing:
> 
> import service
> service.f
> 
> Now "reload(service)" should do what you expect.
> 
> The other way is not to bother with reload. It's not very powerful, only 
> good for the simplest use in the interactive interpreter. Just exit the 
> interpreter and restart it.
> 
> 
> -- 
> Steven

Thank you very much for your excellent explanation!
   I must say that I haven't been using the "from soandso import ..."
formula at all. I thought it might expose names to collision, and why
should I assume the responsibility if I can avoid the problem altogether
using explicit names. If I used the, shall we say, "direct import" this
time it was in an effort to develop a more extensive program. I thought
if a module grows beyond a size that's comfortable to edit, I could just
move select segments to separate files and replace the vacancy with
"from the_respective_segment_module import *", analogous to "#include"
in C.
   The remedy seems to have side-effects that can kill the patient. So
I'll go back to the explicit imports, then. No problem at all. 

Thanking you and the other helpers too

Frederic


0
Frederic
7/9/2010 6:38:43 PM
Frederic Rentsch <anthra.norell@bluewin.ch> wrote:
>
>I develop in an IDLE window.
>
>Module M says 'from service import *'.
>Next I correct a mistake in function 'service.f'.
>Now 'service.f' works fine.
>
>I do 'reload (service); reload (M)'.
>The function 'M.f' still misbehaves.
>
>'print inspect.getsource (service.f)' and
>'print inspect.getsource (M.f)' shows the same 
>corrected code. 
>
>'print service.f' and 'print M.f' show different ids.

Yes. This:

    from service import xxx

is essentially the same as:

    import service
    xxx = service.xxx

At that point, xxx contains a reference to the "service.xxx" object as it
is right now.  When you do a reload, that imports a new version of
"service.xxx", but your global "xxx" object is still bound to the old one.
-- 
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.
0
timr (1409)
7/10/2010 11:24:59 PM
> from m import f
>
> look for module m in the global cache
> if not there, then:
>     search for m.py
>     compile it to a Module object
>     put the Module object in the cache
> look for object named "f" in the Module object
agree

> create a new name "f" in the local namespace
> set the name "f" to cached object
strongly agree

> The important thing to notice is the the name "f" is a local variable. It
> doesn't, and can't, remember that it comes from module m. Reloading m
> can't do anything to f, because the connection is lost.
disagree with 2nd stmt. local 'f' does *remember* (if that's the right
word) where it comes from, because it points to the original obj, as
you said: 'set the name "f" to cached object'

py0.py
======
def f(): ...

>>> from py0 import *
>>> assert f.__module__ == sys.modules['py0'].__name__
>>> assert f.__globals__ is sys.modules['py0'].__dict__

> Now consider that the object "f" that came from m was itself imported
> from another module, "service". Reloading service doesn't help, because
> m.f doesn't know it came from service. Reloading m doesn't help, because
> all that does is run "from service import f" again, and that just fetches
> f from the global cache.
disagree with 2nd stmt, partially disagree with 3rd stmt

reloading 'service' partially helps, since it updates the mod obj
pointed by 'service' in global cache. it needs to be followed by
reloading m, then we have m.f points to the new obj. the important
part is the order of reloading mods

l.py
====
def f(): ...

m.py
====
from l import *

>>> from m import *

at this point, the func obj is referenced by 3 distinct variables with
name 'm'(one in each mod)

>>> assert sys.getrefcount(f) == 4
>>> referrers = gc.get_referrers(f)
>>> mod_dicts = [sys.modules[k].__dict__ for k in sys.modules if k == 'l' or k == 'm' or k == __name__]
>>> for d in mod_dicts:
....	referrers.remove(d)
>>> assert len(referrers) == 0

>>> imp.reload(sys.modules['l'])

now the original func obj is ref'ed by 2 vars, the new func obj is
ref'ed by 1 var

>>> imp.reload(sys.modules['m'])
>>> f = sys.modules['m'].f

now the original func obj is ready to be recollected, the new func obj
is ref'ed by 3 vars

> The simplest, easiest way of dealing with this is not to have to deal
> with it: don't use "from service import f", and ESPECIALLY don't use
> "from service import *". Always use fully-qualified importing:
>
> import service
> service.f
strongly agree

> The other way is not to bother with reload. It's not very powerful, only
> good for the simplest use in the interactive interpreter. Just exit the
> interpreter and restart it.
set, del, reload are useful when it comes to structure manipulation at
runtime, most langs differentiate it, labeling as metaprogramming, py
smashes the diff in an elegant way (oh flattery, i love it) ;)

as their names say, set & del only bind & unbind obj to var, the obj
to bind must be in its bytecode form, in theory one could do it, maybe
thru modules that are categorized as "Python Language Services" in the
lib manual, but it's not practical (e.g. see the last stmt of
types.CodeType.__doc__). this is where 'reload' steps in, and so the
story goes ...
0
kedra
7/11/2010 8:31:07 AM
Reply: