Hi,
  I'm using a custom pickler that replaces any un-pickleable objects (such as sockets or files) with a string representation of them, based on the code from Shane Hathaway here:
http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-some-unpicklable-items

It works most of the time, but when I try to unpickle a Django HttpResponse, I get the following error:
UnpicklingError: NEWOBJ class argument isn't a type object

I have no clue what the error actually means.  If it pickles okay, why should it not be able to unpickle?  Any ideas?

thanks for the help,
imran

Here is my code:

from cPickle import Pickler, Unpickler, UnpicklingError

class FilteredObject:
	def __init__(self, about):
		self.about = about
	def __repr__(self):
		return 'FilteredObject(%s)' % repr(self.about)

class MyPickler(object):
	def __init__(self, file, protocol=2):
		pickler = Pickler(file, protocol)
		pickler.persistent_id = self.persistent_id
		self.dump = pickler.dump
		self.clear_memo = pickler.clear_memo

	def persistent_id(self, obj):
               if not hasattr(obj, '__getstate__') and not isinstance(obj,
			(basestring, bool, int, long, float, complex, tuple, list, set, dict)):
			return ["filtered:%s" % str(obj)]
		else:
			return None

class MyUnpickler(object):
	def __init__(self, file):
		unpickler = Unpickler(file)
		unpickler.persistent_load = self.persistent_load
		self.load = unpickler.load
		self.noload = unpickler.noload

	def persistent_load(self, obj_id):
		if obj_id[0].startswith('filtered:'):
			return FilteredObject(obj_id[0][9:])
		else:
			raise UnpicklingError('Invalid persistent id')

###### serialize to file

f = open('test.txt','wb')
p = MyPickler(f)
p.dump(data)
f.close()

###### unserialize from file

f = open('test.txt','rb')
pickled_data = f.read()
f.seek(0)
u = MyUnpickler(f)
data = u.load()

On Mon, Jul 8, 2013 at 3:27 PM, skunkwerk <skunkwerk@gmail.com> wrote:
>   I'm using a custom pickler that replaces any un-pickleable objects (such as sockets or files) with a string representation of them...
>
> If it pickles okay, why should it not be able to unpickle?  Any ideas?

Generally, the reason something won't pickle is because it won't be
able to be unpickled. So arbitrarily creating a string might allow the
pickle operation to continue, but might well prevent unpickling still.
I don't know, you'd have to play around with it.

ChrisA

skunkwerk <skunkwerk@gmail.com> writes:

> Hi,
>   I'm using a custom pickler that replaces any un-pickleable objects (such as sockets or files) with a string representation of them, based on the code from Shane Hathaway here:
> http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-some-unpicklable-items
>
> It works most of the time, but when I try to unpickle a Django HttpResponse, I get the following error:
> UnpicklingError: NEWOBJ class argument isn't a type object
>
> I have no clue what the error actually means.

The pickling protocol uses a form of bytecode which is executed
during the unpickling to reconstruct the python objects based
on their state found in the pickle alongside the bytecode.

"NEWOBJ" is executed in response to such a bytecode operation.
It expects to get a type as a parameter but in your case,
it gets something else.


> If it pickles okay, why should it not be able to unpickle?  Any ideas?

It is by principle impossible for the pickler to garantee
that an unpickler will later succeed: the pickler does not know
which classes/types are available for the unpickler.

In your special case, the pickler could probably
detect that unpickling will fail - but when
an aim cannot be achieved completely this may provide
motivation to abandon it as a whole - and not put much effort
into a partial achievement.
I have seen many cases where pickling succeeded but
unpickling failed and in principle the pickler could have
already predicted the failure (under the assumption that
the unpickler sees the same classes/types as the pickler).


If it is important for you to get Django HttpResponses
successfully unpickled then you likely need to guide
their pickling process better.

Maybe (as an alternative), you can extract the relevant information
from the "HttpResponse" and pickle that instead of
the response itself?

skunkwerk wrote:

> Hi,
>   I'm using a custom pickler that replaces any un-pickleable objects (such
>   as sockets or files) with a string representation of them, based on the
>   code from Shane Hathaway here:
> http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-
some-unpicklable-items
> 
> It works most of the time, but when I try to unpickle a Django
> HttpResponse, I get the following error: UnpicklingError: NEWOBJ class
> argument isn't a type object
> 
> I have no clue what the error actually means.  If it pickles okay, why
> should it not be able to unpickle?  Any ideas?

A simple way to provoke the error is to rebind the name referring to the 
class of the pickled object:

>>> import cPickle
>>> class A(object): pass
.... 
>>> p = cPickle.dumps(A(), -1)
>>> cPickle.loads(p)
<__main__.A object at 0x7fce7bb58c50>
>>> A = 42
>>> cPickle.loads(p)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cPickle.UnpicklingError: NEWOBJ class argument isn't a type object

You may be doing something to that effect.

On Monday, July 8, 2013 12:45:55 AM UTC-7, Peter Otten wrote:
> skunkwerk wrote:
> 
> 
> 
> > Hi,
> 
> >   I'm using a custom pickler that replaces any un-pickleable objects (such
> 
> >   as sockets or files) with a string representation of them, based on the
> 
> >   code from Shane Hathaway here:
> 
> > http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-
> 
> some-unpicklable-items
> 
> > 
> 
> > It works most of the time, but when I try to unpickle a Django
> 
> > HttpResponse, I get the following error: UnpicklingError: NEWOBJ class
> 
> > argument isn't a type object
> 
> > 
> 
> > I have no clue what the error actually means.  If it pickles okay, why
> 
> > should it not be able to unpickle?  Any ideas?
> 
> 
> 
> A simple way to provoke the error is to rebind the name referring to the 
> 
> class of the pickled object:
> 
> 
> 
> >>> import cPickle
> 
> >>> class A(object): pass
> 
> ... 
> 
> >>> p = cPickle.dumps(A(), -1)
> 
> >>> cPickle.loads(p)
> 
> <__main__.A object at 0x7fce7bb58c50>
> 
> >>> A = 42
> 
> >>> cPickle.loads(p)
> 
> Traceback (most recent call last):
> 
>   File "<stdin>", line 1, in <module>
> 
> cPickle.UnpicklingError: NEWOBJ class argument isn't a type object
> 
> 
> 
> You may be doing something to that effect.

Hey Peter,
  I tried unpickling even from another file with no other code in it, but came up with the same error - so I don't think it's a rebinding issue.

But I got the error to disappear when I removed the "hasattr(obj, '__getstate__')" from this line of code in the persistent_id function:
if not hasattr(obj, '__getstate__') and isinstance(obj,(basestring, bool, int, long, float, complex, tuple, list, set, dict)):
    return ["filtered:%s" % type(obj)]

When I do that, I get a few more FilteredObjects in the result, for things like:
<class 'django.core.handlers.wsgi.WSGIRequest'>
<class 'MySQLdb.connections.Connection'>

I figured these classes must have __getstate__ methods which leads to them being pickled without a persistent_id (it turns out they actually have __repr__ methods).

So these classes get pickled fine, but run into problems when trying to unpickle them.  I understand why ImportErrors would happen if the necessary modules haven't been loaded, but this NEWOBJ error is still kind of mystifying.
I guess I just won't pickle any classes for now, if unpickling them is going to be dicey.

thanks for the help guys,
imran

skunkwerk wrote:

> On Monday, July 8, 2013 12:45:55 AM UTC-7, Peter Otten wrote:
>> skunkwerk wrote:
>> 
>> 
>> 
>> > Hi,
>> 
>> >   I'm using a custom pickler that replaces any un-pickleable objects
>> >   (such
>> 
>> >   as sockets or files) with a string representation of them, based on
>> >   the
>> 
>> >   code from Shane Hathaway here:
>> 
>> > http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-
>> 
>> some-unpicklable-items
>> 
>> > 
>> 
>> > It works most of the time, but when I try to unpickle a Django
>> 
>> > HttpResponse, I get the following error: UnpicklingError: NEWOBJ class
>> 
>> > argument isn't a type object
>> 
>> > 
>> 
>> > I have no clue what the error actually means.  If it pickles okay, why
>> 
>> > should it not be able to unpickle?  Any ideas?
>> 
>> 
>> 
>> A simple way to provoke the error is to rebind the name referring to the
>> 
>> class of the pickled object:
>> 
>> 
>> 
>> >>> import cPickle
>> 
>> >>> class A(object): pass
>> 
>> ...
>> 
>> >>> p = cPickle.dumps(A(), -1)
>> 
>> >>> cPickle.loads(p)
>> 
>> <__main__.A object at 0x7fce7bb58c50>
>> 
>> >>> A = 42
>> 
>> >>> cPickle.loads(p)
>> 
>> Traceback (most recent call last):
>> 
>>   File "<stdin>", line 1, in <module>
>> 
>> cPickle.UnpicklingError: NEWOBJ class argument isn't a type object
>> 
>> 
>> 
>> You may be doing something to that effect.
> 
> Hey Peter,
>   I tried unpickling even from another file with no other code in it, but
>   came up with the same error - so I don't think it's a rebinding issue.
> 
> But I got the error to disappear when I removed the "hasattr(obj,
> '__getstate__')" from this line of code in the persistent_id function: if
> not hasattr(obj, '__getstate__') and isinstance(obj,(basestring, bool,
> int, long, float, complex, tuple, list, set, dict)):
>     return ["filtered:%s" % type(obj)]
> 
> When I do that, I get a few more FilteredObjects in the result, for things
> like: <class 'django.core.handlers.wsgi.WSGIRequest'>
> <class 'MySQLdb.connections.Connection'>
> 
> I figured these classes must have __getstate__ methods which leads to them
> being pickled without a persistent_id (it turns out they actually have
> __repr__ methods).
> 
> So these classes get pickled fine, but run into problems when trying to
> unpickle them.  I understand why ImportErrors would happen if the
> necessary modules haven't been loaded, but this NEWOBJ error is still kind
> of mystifying. I guess I just won't pickle any classes for now, if
> unpickling them is going to be dicey.
> 
> thanks for the help guys,
> imran

Maybe you can find the problem by temporarily switching from cPickle to the 
pickle module which produces a slightly more helpful traceback:

>>> import cPickle, pickle
>>> class A(object): pass
.... 
>>> p = cPickle.dumps(A(), -1)
>>> A = 42
>>> cPickle.loads(p)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cPickle.UnpicklingError: NEWOBJ class argument isn't a type object
>>> pickle.loads(p)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1382, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1083, in load_newobj
    obj = cls.__new__(cls, *args)
TypeError: int.__new__(X): X is not a type object (int)