Hi,

>>> snapdir = './mango.txt_snaps'
>>> snap_cnt = 1
>>> block = 0
>>> import os
>>> os.chdir('/mnt/gfs_local')
>>> snap = open(snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block)),'r')
>>> snap.read()
'dfdfdgagdfgdf\ngdgfadgagadg\nagafg\n\nfs\nf\nsadf\n\nsdfsdfsadf\n'
>>> snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block))
'./mango.txt_snaps/snap1/0'

The above code works fine and it reads the whole file till EOF. But
when this method is used in a different scenario the file is not read
completely. I'll post the code that read only some part of the file...

self.snap = open(self.snapdir + '/snap%d/%d' % (self.snap_cnt,
block),'r') ## opens /mnt/gfs_local/mango.txt_snaps/snap1/0
self.snap.seek(off%4096) ## seeks to 0 in this case
bend = 4096-(off%4096) ## 4096 in this case
if length-bend <= 0:    ## true in this case as length is 4096
	tf.writelines("returned \n")
	data = self.snap.read(length)
	self.snap.close()
	break

the output data is supposed to read the whole fie but it only reads a
part of it. Why is it encountering an early EOF ?

Sreejith K <sreejithemk@gmail.com> wrote:
> Hi,
> 
> >>> snapdir = './mango.txt_snaps'
> >>> snap_cnt = 1
> >>> block = 0
> >>> import os
> >>> os.chdir('/mnt/gfs_local')
> >>> snap = open(snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block)),'r')
> >>> snap.read()
> 'dfdfdgagdfgdf\ngdgfadgagadg\nagafg\n\nfs\nf\nsadf\n\nsdfsdfsadf\n'
> >>> snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block))
> './mango.txt_snaps/snap1/0'
> 
> The above code works fine and it reads the whole file till EOF. But
> when this method is used in a different scenario the file is not read
> completely. I'll post the code that read only some part of the file...
> 
> self.snap = open(self.snapdir + '/snap%d/%d' % (self.snap_cnt,
> block),'r') ## opens /mnt/gfs_local/mango.txt_snaps/snap1/0
> self.snap.seek(off%4096) ## seeks to 0 in this case
> bend = 4096-(off%4096) ## 4096 in this case
> if length-bend <= 0:    ## true in this case as length is 4096
> 	tf.writelines("returned \n")
> 	data = self.snap.read(length)
> 	self.snap.close()
> 	break
> 
> the output data is supposed to read the whole fie but it only reads a
> part of it. Why is it encountering an early EOF ?

It's not.  In the second case you told it to read only 4096 bytes.  You
might want to read the docs for the 'read' method, paying particular
attention to the optional argument and its meaning.

--
R. David Murray           http://www.bitdance.com

On Mar 20, 4:43=A0pm, "R. David Murray" <rdmur...@bitdance.com> wrote:
> Sreejith K <sreejith...@gmail.com> wrote:
> > Hi,
>
> > >>> snapdir =3D './mango.txt_snaps'
> > >>> snap_cnt =3D 1
> > >>> block =3D 0
> > >>> import os
> > >>> os.chdir('/mnt/gfs_local')
> > >>> snap =3D open(snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block)=
),'r')
> > >>> snap.read()
> > 'dfdfdgagdfgdf\ngdgfadgagadg\nagafg\n\nfs\nf\nsadf\n\nsdfsdfsadf\n'
> > >>> snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block))
> > './mango.txt_snaps/snap1/0'
>
> > The above code works fine and it reads the whole file till EOF. But
> > when this method is used in a different scenario the file is not read
> > completely. I'll post the code that read only some part of the file...
>
> > self.snap =3D open(self.snapdir + '/snap%d/%d' % (self.snap_cnt,
> > block),'r') ## opens /mnt/gfs_local/mango.txt_snaps/snap1/0
> > self.snap.seek(off%4096) ## seeks to 0 in this case
> > bend =3D 4096-(off%4096) ## 4096 in this case
> > if length-bend <=3D 0: =A0 =A0## true in this case as length is 4096
> > =A0 =A0tf.writelines("returned \n")
> > =A0 =A0data =3D self.snap.read(length)
> > =A0 =A0self.snap.close()
> > =A0 =A0break
>
> > the output data is supposed to read the whole fie but it only reads a
> > part of it. Why is it encountering an early EOF ?
>
> It's not. =A0In the second case you told it to read only 4096 bytes. =A0Y=
ou
> might want to read the docs for the 'read' method, paying particular
> attention to the optional argument and its meaning.
>
> --
> R. David Murray =A0 =A0 =A0 =A0 =A0http://www.bitdance.com

I'm using the above codes in a pthon-fuse's file class's read
function. The offset and length are 0 and 4096 respectively for my
test inputs. When I open a file and read the 4096 bytes from offset,
only a few lines are printed, not the whole file. Actually the file is
only a few bytes. But when I tried reading from the Interactive mode
of python it gave the whole file.

Is there any problem using read() method in fuse-python ?

Also statements like break and continue behaves weirdly in fuse
functions. Any help is appreciated....

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--8323328-999735971-1237562442=:26433
Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed
Content-Transfer-Encoding: QUOTED-PRINTABLE

On Fri, 20 Mar 2009 at 07:09, Sreejith K wrote:
> On Mar 20, 4:43=A0pm, "R. David Murray" <rdmur...@bitdance.com> wrote:
>> Sreejith K <sreejith...@gmail.com> wrote:
>>> Hi,
>>
>>>>>> snapdir =3D './mango.txt_snaps'
>>>>>> snap_cnt =3D 1
>>>>>> block =3D 0
>>>>>> import os
>>>>>> os.chdir('/mnt/gfs_local')
>>>>>> snap =3D open(snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block))=
,'r')
>>>>>> snap.read()
>>> 'dfdfdgagdfgdf\ngdgfadgagadg\nagafg\n\nfs\nf\nsadf\n\nsdfsdfsadf\n'
>>>>>> snapdir + '/snap%s/%s' % (repr(snap_cnt), repr(block))
>>> './mango.txt_snaps/snap1/0'
>>
>>> The above code works fine and it reads the wholefiletill EOF. But
>>> when this method is used in a different scenario thefileis notread
>>> completely. I'll post the code thatreadonly some part of thefile...
>>
>>> self.snap =3D open(self.snapdir + '/snap%d/%d' % (self.snap_cnt,
>>> block),'r') ## opens /mnt/gfs_local/mango.txt_snaps/snap1/0
>>> self.snap.seek(off%4096) ## seeks to 0 in this case
>>> bend =3D 4096-(off%4096) ## 4096 in this case
>>> if length-bend <=3D 0: =A0 =A0## true in this case as length is 4096
>>> =A0 =A0tf.writelines("returned \n")
>>> =A0 =A0data =3D self.snap.read(length)
>>> =A0 =A0self.snap.close()
>>> =A0 =A0break
>>
>>> the output data is supposed toreadthe whole fie but it only reads a
>>> part of it. Why is it encountering an early EOF ?
>>
>> It's not. =A0In the second case you told it toreadonly 4096 bytes. =A0Yo=
u
>> might want toreadthe docs for the 'read' method, paying particular
>> attention to the optional argument and its meaning.
>>
>> --
>> R. David Murray =A0 =A0 =A0 =A0 =A0http://www.bitdance.com
>
> Thanks for the reply,
> Actually the file is only few bytes and file.read() and file.read
> (4096) will give the same result, i.e the whole file. But in my case
> its not happening (using it in python-fuse file class).. Any other
> ideas ?

Not offhand.  If it were me I'd start playing with parameters and moving
things around, trying to find additional clues.  See if calling read
without the argument in the second case works, start stripping the second
case down until it starts working (even if wrongly for the ultimate goal),
and things like that.

--
R. David Murray           http://www.bitdance.com
--8323328-999735971-1237562442=:26433--

Sreejith K wrote:

> I'm using the above codes in a pthon-fuse's file class's read
> function. The offset and length are 0 and 4096 respectively for my
> test inputs. When I open a file and read the 4096 bytes from offset,
> only a few lines are printed, not the whole file. Actually the file is
> only a few bytes. But when I tried reading from the Interactive mode
> of python it gave the whole file.
> 
> Is there any problem using read() method in fuse-python ?
> 
> Also statements like break and continue behaves weirdly in fuse
> functions. Any help is appreciated....

I had never heard of fuse until today.  I get the impression that 
working with fuse is not the same as working with the stock os 
filesystem, so you might have mentioned this from the beginning;-).  I 
suggest you direct these questions to python-fuse and fuse-python folk.

On Fri, 20 Mar 2009 07:03:35 -0700, Sreejith K wrote:
> I'm using the above codes in a pthon-fuse's file class's read function.
> The offset and length are 0 and 4096 respectively for my test inputs.
> When I open a file and read the 4096 bytes from offset, only a few lines
> are printed, not the whole file. Actually the file is only a few bytes.
> But when I tried reading from the Interactive mode of python it gave the
> whole file.

Your example doesn't show what you are doing with "data" after you've 
read it. Presumably you're outputting it somehow, which is where you see 
that it doesn't contain the whole file. But are you sure the problem is 
in the reading, and not in the outputting? Could you show the section of 
the code where you output "data"?

On Mar 21, 12:58=A0am, I V <ivle...@gmail.com> wrote:
> On Fri, 20 Mar 2009 07:03:35 -0700, Sreejith K wrote:
> > I'm using the above codes in a pthon-fuse's file class's read function.
> > The offset and length are 0 and 4096 respectively for my test inputs.
> > When I open a file and read the 4096 bytes from offset, only a few line=
s
> > are printed, not the whole file. Actually the file is only a few bytes.
> > But when I tried reading from the Interactive mode of python it gave th=
e
> > whole file.
>
> Your example doesn't show what you are doing with "data" after you've
> read it. Presumably you're outputting it somehow, which is where you see
> that it doesn't contain the whole file. But are you sure the problem is
> in the reading, and not in the outputting? Could you show the section of
> the code where you output "data"?
Thanks,

I think there is no use in showing the whole code (which is a
filesystem implementation using python-fuse), but if you want it I'll
post it, see the next post :)

Forget about the 'data' variable (I'm just returning it at the end of
the function). But even if I'm returning self.snap.read(length), it
doesn't read length (if length is larger than the size of the file
then till EOF) bytes from the file.

But this only happens when the second code samples (they're used in
fuse-python classes) and not in the first example. In the normal
interactive python shell the same procedure returns the whole file
(using read() or read(4096) which is a particular case)...

I'm sure, In my case, self.snap is opening the file ./mango.txt_snaps/
snap1/0 and off and length variables are 0 and 4096 respectively. The
file ./mango.txt_snaps/snap1/0 is only 56 bytes with 9 lines of text.
But either read(4096)/read() in the second example reads/returns only
7 lines. But this happens only in when using it in a fuse-python file
class's read function. In normal interactive mode it reads the whole
file. Hope you understand the problem now. I'm very sorry that I
couldn't explain the problem very well.

	class MedusaFile(object):
		def __init__(self, path, flags, *mode):
			global METHOD
			global NORMAL
			global SNAP
			global FRESH_SNAP
			self.path = path
			tf.writelines("File initiating..\n")
			self.file = os.fdopen(os.open("." + path, flags, *mode),flag2mode
(flags))
			self.fd = self.file.fileno()
			curdir = GetDirPath('.' + path)
			self.snapdir = '.' + path + '_snaps'
			self.snap_cnt = 0
			if os.path.exists(self.snapdir):
				self.snap_cnt = len(os.listdir(self.snapdir))
				METHOD = SNAP
			elif FRESH_SNAP == 1:
				self.snap_cnt += 1
				os.mkdir(self.snapdir)
				os.mkdir(self.snapdir+'/snap%s' % repr(self.snap_cnt))
				METHOD = SNAP
				FRESH_SNAP = 0
			tf.writelines("File initiated..\n")
		def read(self, length, offset):
			global METHOD
			global NORMAL
			global SNAP
			tf.writelines("Read length: %d offset: %d\n" % (length,offset))
			blk_num = offset/4096
			no_blks = length/4096 + 1
			blocks = []
			block_read = False
			## Read form the Base File (Snap 1)
			if METHOD == NORMAL:
				self.file.seek(offset)
				tf.writelines("Normal read\n")
				return self.file.read(length)
			## Read blocks from the snapshots
			else:
				snap_list = range(self.snap_cnt)
				rev_snap_list = snap_list
				rev_snap_list.reverse()
				for i in snap_list:
					blocks.append(os.listdir(self.snapdir+'/snap%s' % repr(i+1))) ##
list of blocks in the current snapshot
				off = offset
				bend = 0
				data = ''
				for block in range(blk_num,blk_num+no_blks): ## loop through the
blocks which are to be read
					for i in rev_snap_list: ## loop through snapshots (starting from
latest snap)
						tf.writelines('Snapshot %d opened..\n' % i)
						if repr(block) in blocks[i]:  ## Check if it is in the snapshot
							tf.writelines("Snap read\n")
							self.snap = open(self.snapdir + '/snap%d/%d' % (i+1,
block),'r')
							self.snap.seek(off%4096)
							bend = 4096-(off%4096)
							block_read = True
							if length-bend <= 0: ## if only a part of block file is to be
read (i.e. not till the end of block file)
								tf.writelines("Partial read from snap \n")
								data = self.snap.read(length)
								self.snap.close()
								break
							tf.writelines("Full block read from snap\n")
							data += self.snap.read(bend)
							length -= bend
							off = 4096
							self.snap.close()
							break
						tf.writelines("Block not in snap\n")
					if block_read:
						block_read = False
						continue
					## otherwise the block should be in the base file itself. So,
read from there
					tf.writelines("Reading from Base File\n")
					self.file.seek(block*4096 + off%4096)
					bend = 4096-(off%4096)
					if length-bend <= 0: ## if only a part of a block is to be read
(not till the end of the block)
						data = self.file.read(length)
						break
					data += self.file.read(bend)
					length -= bend
					off = 4096
				return data

This is the filesystem class for files. Whenever a read occurs an
instance is created and read function is called. In my example when
accessing a file named 'mango.txt' it checks for mango.txt_snaps/snap1
dirctory and open file '0' as self.snap. But the read() returns (i.e
data) a small part....

Almost all the code worked weird in this example. Apart from read(),
the break and continue also works weird. When opening the file
'mango.txt' the following output is written by tf (an output file).
Here METHOD is not NORMAL, self.snap_cnt is 1, blocks is [['0']]

File initiating..
File initiated..
Read length: 4096 offset: 0
Snapshot 0 opened..
Snap read
Partial read from snap
Snapshot 0 opened..
Block not in snap
Reading from Base File

See the weirdness of continue and break here ?(loop was supposed to
loop only once as rev_snap_list contains 0 only)

Sreejith K <sreejithemk@gmail.com> wrote:
> 					tf.writelines("Reading from Base File\n")
> 					self.file.seek(block*4096 + off%4096)
> 					bend = 4096-(off%4096)
> 					if length-bend <= 0: ## if only a part of a block is to be read
> (not till the end of the block)
> 						data = self.file.read(length)
> 						break
> 					data += self.file.read(bend)
> 					length -= bend
> 					off = 4096
> 				return data
> 
> This is the filesystem class for files. Whenever a read occurs an
> instance is created and read function is called. In my example when
> accessing a file named 'mango.txt' it checks for mango.txt_snaps/snap1
> dirctory and open file '0' as self.snap. But the read() returns (i.e
> data) a small part....
> 
> Almost all the code worked weird in this example. Apart from read(),
> the break and continue also works weird. When opening the file
> 'mango.txt' the following output is written by tf (an output file).
> Here METHOD is not NORMAL, self.snap_cnt is 1, blocks is [['0']]
> 
> File initiating..
> File initiated..
> Read length: 4096 offset: 0
> Snapshot 0 opened..
> Snap read
> Partial read from snap
> Snapshot 0 opened..
> Block not in snap
> Reading from Base File
> 
> See the weirdness of continue and break here ?(loop was supposed to
> loop only once as rev_snap_list contains 0 only)

I'm not going to look at all this code now...it looks way too complicated
and in need of some serious refactoring :)

But a couple of on-point comments:

How do you know rev_snap_list contains only 0?  You didn't log it.

Same for the read.  How do you know the read didn't read the whole
file?  You didn't log it.

Both your statements might be true, but until you show the logging
output proving it, you don't _know_ that your assumptions are true.

--
R. David Murray           http://www.bitdance.com

On Mar 21, 10:54=A0am, "R. David Murray" <rdmur...@bitdance.com> wrote:
> Sreejith K <sreejith...@gmail.com> wrote:
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
tf.writelines("Reading from Base File\n")
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
self.file.seek(block*4096 + off%4096)
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
bend =3D 4096-(off%4096)
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
if length-bend <=3D 0: ## if only a part of a block is to be read
> > (not till the end of the block)
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 =A0 =A0 =A0data =3D self.file.read(length)
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 =A0 =A0 =A0break
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
data +=3D self.file.read(bend)
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
length -=3D bend
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
off =3D 4096
> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return data
>
> > This is the filesystem class for files. Whenever a read occurs an
> > instance is created and read function is called. In my example when
> > accessing a file named 'mango.txt' it checks for mango.txt_snaps/snap1
> > dirctory and open file '0' as self.snap. But the read() returns (i.e
> > data) a small part....
>
> > Almost all the code worked weird in this example. Apart from read(),
> > the break and continue also works weird. When opening the file
> > 'mango.txt' the following output is written by tf (an output file).
> > Here METHOD is not NORMAL, self.snap_cnt is 1, blocks is [['0']]
>
> > File initiating..
> > File initiated..
> > Read length: 4096 offset: 0
> > Snapshot 0 opened..
> > Snap read
> > Partial read from snap
> > Snapshot 0 opened..
> > Block not in snap
> > Reading from Base File
>
> > See the weirdness of continue and break here ?(loop was supposed to
> > loop only once as rev_snap_list contains 0 only)
>
> I'm not going to look at all this code now...it looks way too complicated
> and in need of some serious refactoring :)
>
> But a couple of on-point comments:
>
> How do you know rev_snap_list contains only 0? =A0You didn't log it.
>
> Same for the read. =A0How do you know the read didn't read the whole
> file? =A0You didn't log it.
>
> Both your statements might be true, but until you show the logging
> output proving it, you don't _know_ that your assumptions are true.
>
> --
> R. David Murray =A0 =A0 =A0 =A0 =A0http://www.bitdance.com

Thanks for your comments.

As I said, self.snap_cnt is 1 then

>>> snap_list =3D range(1)
>>> rev_snap_list =3D snap_list
>>> rev_snap_list
[0]
>>> rev_snap_list.reverse()
>>> rev_snap_list
[0]

So I thought there is no need to log it. In the above code, as break
and continue doesn't work as I expected the read() is called twice
("Partial read from snap" and "reading from base file") the output is
a combination of the 2 reads (so it can't be read using 'less'). But
the expected result was "Partial read from snap" i.e read(4096) and
break from the loop. Instead after this break, the loop is again
started ("Snapshot 0 opened.. "). I have no idea what went wrong here.
But when I comment out block_read =3D False before the continue
statement the log displays the following and the file is read
partially....

File initiating..
File initiated..
Read length: 4096 offset: 0
Snapshot 0 opened..
Snap read
Partial read from snap
Snapshot 0 opened..
Block not in snap

The break and continue problem was actually my own mistake. I wrote
no_blks = length/4096 + 1, so the loop actually executes twice. Sorry
for my idiotic mistake....

But the read() problem still persists.....

Thanks..

Sreejith K wrote:
> The break and continue problem was actually my own mistake. I wrote
> no_blks = length/4096 + 1, so the loop actually executes twice. Sorry
> for my idiotic mistake....
> 
That's good!

> But the read() problem still persists.....
> 
Try and write an example that shows the problem in fifteen lines or
less. Much easier for us to focus on the issue that way.

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
Holden Web LLC                 http://www.holdenweb.com/
Want to know? Come to PyCon - soon! http://us.pycon.org/

> Try and write an example that shows the problem in fifteen lines or
> less. Much easier for us to focus on the issue that way.

import os
def read(length, offset):
	os.chdir('/mnt/gfs_local/')
	snap = open('mango.txt_snaps/snap1/0','r')
	snap.seek(offset)
	data = snap.read(length)
	print data

read(4096,0)

This code shows what actually happens inside the code I've written.
This prints the 4096 bytes from the file '0' which is only 654 bytes.
When we run the code we get the whole file. That's right. I also get
it. But when this read() function becomes the file class read()
function in fuse, the data printed is not the whole but only a few
lines from the beginning. I usually use less to read a file, when
'less'ing a file (whose size is less than 4096bytes) a call to read
(0,4096) is made and data is returned. 'less' use this data returned
by my fuse read() function to display its contents. But it was
supposed to be the whole lines in the file like the example, but its
not.... This is the problem I'm facing. Did I do something wrong here ?

> Try and write an example that shows the problem in fifteen lines or
> less. Much easier for us to focus on the issue that way.

import os
def read(length, offset):
	os.chdir('/mnt/gfs_local/')
	snap = open('mango.txt_snaps/snap1/0','r')
	snap.seek(offset)
	data = snap.read(length)
	print data

read(4096,0)

This code shows what actually happens inside the code I've written.
This prints the 4096 bytes from the file '0' which is only 654 bytes.
When we run the code we get the whole file. That's right. I also get
it. But when this read() function becomes the file class read()
function in fuse, the data printed is not the whole but only a few
lines from the beginning. I usually use less to read a file, when
'less'ing a file (whose size is less than 4096bytes) a call to read
(0,4096) is made and data is returned. 'less' use this data returned
by my fuse read() function to display its contents. But it was
supposed to be the whole lines in the file like the example, but its
not.... This is the problem I'm facing. Did I do something wrong here ?

> Try and write an example that shows the problem in fifteen lines or
> less. Much easier for us to focus on the issue that way.

import os
def read(length, offset):
	os.chdir('/mnt/gfs_local/')
	snap = open('mango.txt_snaps/snap1/0','r')
	snap.seek(offset)
	data = snap.read(length)
	print data

read(4096,0)

This code shows what actually happens inside the code I've written.
This prints the 4096 bytes from the file '0' which is only 654 bytes.
When we run the code we get the whole file. That's right. I also get
it. But when this read() function becomes the file class read()
function in fuse, the data printed is not the whole but only a few
lines from the beginning. I usually use less to read a file, when
'less'ing a file (whose size is less than 4096bytes) a call to read
(0,4096) is made and data is returned. 'less' use this data returned
by my fuse read() function to display its contents. But it was
supposed to be the whole lines in the file like the example, but its
not.... This is the problem I'm facing. Did I do something wrong here ?

Sreejith K <sreejithemk@gmail.com> wrote:
> > Try and write an example that shows the problem in fifteen lines or
> > less. Much easier for us to focus on the issue that way.
> 
> import os
> def read(length, offset):
> 	os.chdir('/mnt/gfs_local/')
> 	snap = open('mango.txt_snaps/snap1/0','r')
> 	snap.seek(offset)
> 	data = snap.read(length)
> 	print data
> 
> read(4096,0)
> 
> This code shows what actually happens inside the code I've written.
> This prints the 4096 bytes from the file '0' which is only 654 bytes.
> When we run the code we get the whole file. That's right. I also get
> it. But when this read() function becomes the file class read()
> function in fuse, the data printed is not the whole but only a few
> lines from the beginning. I usually use less to read a file, when
> 'less'ing a file (whose size is less than 4096bytes) a call to read
> (0,4096) is made and data is returned. 'less' use this data returned
> by my fuse read() function to display its contents. But it was
> supposed to be the whole lines in the file like the example, but its
> not.... This is the problem I'm facing. Did I do something wrong here ?

If I'm understanding you correctly, you are saying that when you use
this function as the fuse read function you don't get the whole file,
and you are verifying this by using 'less' to read the 'file' exposed
by fuse.  Correct?

So you still have not decoupled the python read from the fuse read in
your debugging.  You are focused on the fact that the python read "must
be failing", yet you still (as far as you have told us) not _proven_
that by logging the value returned from the read.  Until you do that,
you can't even be sure where your problem is.

If you have done it, show us the logging output, please.

--
R. David Murray           http://www.bitdance.com

Sreejith K wrote:
>> Try and write an example that shows the problem in fifteen lines or
>> less. Much easier for us to focus on the issue that way.
> 
> import os
> def read(length, offset):
> 	os.chdir('/mnt/gfs_local/')
> 	snap = open('mango.txt_snaps/snap1/0','r')
> 	snap.seek(offset)
> 	data = snap.read(length)
> 	print data
> 
> read(4096,0)
> 
> This code shows what actually happens inside the code I've written.
> This prints the 4096 bytes from the file '0' which is only 654 bytes.
> When we run the code we get the whole file. That's right. I also get
> it. But when this read() function becomes the file class read()
> function in fuse, the data printed is not the whole but only a few
> lines from the beginning.

This is confusing. I presume you to mean that when you make this
function a method of some class it stops operating correctly?

But I am not sure.

I am still struggling to understand your problem. Sorry,it's just a
language thing. If we take our time we will understand each other in the
end.

regards
 Steve
> I usually use less to read a file, when
> 'less'ing a file (whose size is less than 4096bytes) a call to read
> (0,4096) is made and data is returned. 'less' use this data returned
> by my fuse read() function to display its contents. But it was
> supposed to be the whole lines in the file like the example, but its
> not.... This is the problem I'm facing. Did I do something wrong here ?
> --
> http://mail.python.org/mailman/listinfo/python-list
> 


-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
Holden Web LLC                 http://www.holdenweb.com/
Want to know? Come to PyCon - soon! http://us.pycon.org/

Steve Holden <steve@holdenweb.com> wrote:
> Sreejith K wrote:
> >> Try and write an example that shows the problem in fifteen lines or
> >> less. Much easier for us to focus on the issue that way.
> > 
> > import os
> > def read(length, offset):
> > 	os.chdir('/mnt/gfs_local/')
> > 	snap = open('mango.txt_snaps/snap1/0','r')
> > 	snap.seek(offset)
> > 	data = snap.read(length)
> > 	print data
> > 
> > read(4096,0)
> > 
> > This code shows what actually happens inside the code I've written.
> > This prints the 4096 bytes from the file '0' which is only 654 bytes.
> > When we run the code we get the whole file. That's right. I also get
> > it. But when this read() function becomes the file class read()
> > function in fuse, the data printed is not the whole but only a few
> > lines from the beginning.
> 
> This is confusing. I presume you to mean that when you make this
> function a method of some class it stops operating correctly?
> 
> But I am not sure.
> 
> I am still struggling to understand your problem. Sorry,it's just a
> language thing. If we take our time we will understand each other in the
> end.

You may be asking this question for pedagogical reasons, Steve, but
in case not...the OP is apparently doing a 'less xxxx' where xxxx is
the name of a file in a fuse filesystem (that is, a mounted filesystem
whose back end is some application code written by the OP).  So when
the OP runs less, several calls get made to fuse, which passes them to
fuse-python, which calls methods on the OP's python class.  He is looking
in particular at the 'read' call, which happens after 'less' has opened
the file and wants to read a block (apparently either less or fuse is
asking for the first 4096 bytes of the file).  At that point his 'read'
method above is called.  But based on what he's told us it appears his
conclusion that the 'snap.read(length)' call is not returning the whole
file is based on the fact that less is only showing him part of the file.
There are several steps between that 'snap.read' and less displaying on
the terminal whatever bytes it got back from its read call in whatever
way it is less chooses to display them....

--
R. David Murray           http://www.bitdance.com

En Mon, 23 Mar 2009 21:37:14 -0300, R. David Murray  
<rdmurray@bitdance.com> escribi�:
> Steve Holden <steve@holdenweb.com> wrote:
>> Sreejith K wrote:
>> >> Try and write an example that shows the problem in fifteen lines or
>> >> less. Much easier for us to focus on the issue that way.
>> >
>> > import os
>> > def read(length, offset):
>> > 	os.chdir('/mnt/gfs_local/')
>> > 	snap = open('mango.txt_snaps/snap1/0','r')
>> > 	snap.seek(offset)
>> > 	data = snap.read(length)
>> > 	print data
>> >
>> > read(4096,0)
>> >
>> > This code shows what actually happens inside the code I've written.
>> > This prints the 4096 bytes from the file '0' which is only 654 bytes.
>> > When we run the code we get the whole file. That's right. I also get
>> > it. But when this read() function becomes the file class read()
>> > function in fuse, the data printed is not the whole but only a few
>> > lines from the beginning.
>>
>> This is confusing. I presume you to mean that when you make this
>> function a method of some class it stops operating correctly?
>>
>> But I am not sure.
>>
>> I am still struggling to understand your problem. Sorry,it's just a
>> language thing. If we take our time we will understand each other in the
>> end.
>
> You may be asking this question for pedagogical reasons, Steve, but
> in case not...the OP is apparently doing a 'less xxxx' where xxxx is
> the name of a file in a fuse filesystem (that is, a mounted filesystem
> whose back end is some application code written by the OP).
[...]
> There are several steps between that 'snap.read' and less displaying on
> the terminal whatever bytes it got back from its read call in whatever
> way it is less chooses to display them....

And that's why everyone is asking for a *real* log. Assumptions like "foo  
must be 0 here" aren't enough. One needs *evidence*: a log file showing  
the value of "foo" right when it is used. Then, one can begin to infer  
what happens -- first step would be to determine *which* layer is (or is  
not) responsible for the misbehavior.

In this case, I'd like to see file.tell(), the requested size and the  
returned data length, *right*at*the*read()*call*.

-- 
Gabriel Genellina

On Mar 24, 7:15=A0am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
wrote:
> En Mon, 23 Mar 2009 21:37:14 -0300, R. David Murray =A0
> <rdmur...@bitdance.com> escribi=F3:
>
>
>
> > Steve Holden <st...@holdenweb.com> wrote:
> >> Sreejith K wrote:
> >> >> Try and write an example that shows the problem in fifteen lines or
> >> >> less. Much easier for us to focus on the issue that way.
>
> >> > import os
> >> > def read(length, offset):
> >> > =A0 os.chdir('/mnt/gfs_local/')
> >> > =A0 snap =3D open('mango.txt_snaps/snap1/0','r')
> >> > =A0 snap.seek(offset)
> >> > =A0 data =3D snap.read(length)
> >> > =A0 print data
>
> >> > read(4096,0)
>
> >> > This code shows what actually happens inside the code I've written.
> >> > This prints the 4096 bytes from the file '0' which is only 654 bytes=
..
> >> > When we run the code we get the whole file. That's right. I also get
> >> > it. But when this read() function becomes the file class read()
> >> > function in fuse, the data printed is not the whole but only a few
> >> > lines from the beginning.
>
> >> This is confusing. I presume you to mean that when you make this
> >> function a method of some class it stops operating correctly?
>
> >> But I am not sure.
>
> >> I am still struggling to understand your problem. Sorry,it's just a
> >> language thing. If we take our time we will understand each other in t=
he
> >> end.
>
> > You may be asking this question for pedagogical reasons, Steve, but
> > in case not...the OP is apparently doing a 'less xxxx' where xxxx is
> > the name of a file in a fuse filesystem (that is, a mounted filesystem
> > whose back end is some application code written by the OP).
> [...]
> > There are several steps between that 'snap.read' and less displaying on
> > the terminal whatever bytes it got back from its read call in whatever
> > way it is less chooses to display them....
>
> And that's why everyone is asking for a *real* log. Assumptions like "foo=
 =A0
> must be 0 here" aren't enough. One needs *evidence*: a log file showing =
=A0
> the value of "foo" right when it is used. Then, one can begin to infer =
=A0
> what happens -- first step would be to determine *which* layer is (or is =
=A0
> not) responsible for the misbehavior.
>
> In this case, I'd like to see file.tell(), the requested size and the =A0
> returned data length, *right*at*the*read()*call*.
>
> --
> Gabriel Genellina

R. David Murray understood the problem very well. As you've said I
logged the data returned and see that it actually the complete file.
But the problem is when 'less'ing only two lines are displayed. So its
not regarding the read() of python. Usually in fuse filesystems (as in
some examples), when some file read occurs fuse catch it and calls the
python-fuse's read(length, offset) function. For small files (when we
read using 'less'), this will be usually the first block i.e 4096 with
offset 0. We return what we read from the read() method of fuse-
python. But when I return the data I read, a 'less' operation in my
fuse filesystem shows some lines only, even if the returned data is
the whole file.

In my implementation of fuse-filesystem when a read is called, instead
of returning the data read from the original file, I return the data
read from another file ('0') which resides in <original-file>__snaps/
snap1 directory (I use this directory to store the blocks of original
files when write occurs. So each file here would be 4096 bytes). I'm
doing this because I want to make snapshots of files so that I can
restore the older file easily. The problem occurs when reading this
file and returning the read data.

Some flush/release functions are there in fuse to properly close the
opened file. When reading (less) the original file without snapshots,
there is no issue. But when reading the snapshot instead, the problem
occurs. I open the snapshot file with the same modes as the original
file. Is there anything I should do after read() like the flush() as
for the original file ? I tried it, but no success...

Log when reading from snapshot
=3D=3D=3D=3D=3D=3D=3D
Read length: 4096 offset: 0
Snapshot 0 opened..
Snap read
=3D=3D=3DData Begin=3D=3D=3D
Getting started -- pdb.set_trace()

To start, I'll show you the very simplest way to use the Python
debugger.

   1. Let's start with a simple program, epdb1.py.

              # epdb1.py -- experiment with the Python debugger, pdb
              a =3D "aaa"
              b =3D "bbb"
              c =3D "ccc"
              final =3D a + b + c
              print final


   2. Insert the following statement at the beginning of your Python
program. This statement imports the Python debugger module, pdb.

          import pdb

   3. Now find a spot where you would like tracing to begin, and
insert the following code:

          pdb.set_trace()
=3D=3D=3DData End=3D=3D=3D=3D
snap.tell(): 654
data size : 654
Original file flushed
Original file closed

data is the whole file, but 'less' gives only the two lines...

On Mar 24, 7:59=A0am, Sreejith K <sreejith...@gmail.com> wrote:
....
> data is the whole file, but 'less' gives only the two lines...

From this statement (that you are using less), it appears that you are
redirecting sys.stdout to a file or similar - if that is the case, you
may need to flush or close the output file before it picks up any
changes.

On Mar 24, 2:12=A0pm, Ant <ant...@gmail.com> wrote:
> On Mar 24, 7:59=A0am, Sreejith K <sreejith...@gmail.com> wrote:
> ...
>
> > data is the whole file, but 'less' gives only the two lines...
>
> From this statement (that you are using less), it appears that you are
> redirecting sys.stdout to a file or similar - if that is the case, you
> may need to flush or close the output file before it picks up any
> changes.

Yes, I did try a flush() and close on the file which is read, but the
result is the same. I think when a read comes in fuse redirecting the
actual path (original file) to some other file for reading causes some
issues like this. It would be really helpful if someone help me clear
this issue. The getattr() calls are actually called upon the original
file (doing an os.lstat()). Does this make any effect on the 'less'
output ?

Sreejith K <sreejithemk@gmail.com> wrote:
> On Mar 24, 2:12 pm, Ant <ant...@gmail.com> wrote:
> > On Mar 24, 7:59 am, Sreejith K <sreejith...@gmail.com> wrote:
> > ...
> >
> > > data is the whole file, but 'less' gives only the two lines...
> >
> > From this statement (that you are using less), it appears that you are
> > redirecting sys.stdout to a file or similar - if that is the case, you
> > may need to flush or close the output file before it picks up any
> > changes.

It's not a redirect to a file.  Fuse calls the 'read' function on the
class, the read function does a 'return' of the data, and fuse passes
the data up through the OS layer to be the result of the 'read' call
made by less.

If you don't know what a fuse file system is, this all gets very confusing :)

> Yes, I did try a flush() and close on the file which is read, but the
> result is the same. I think when a read comes in fuse redirecting the
> actual path (original file) to some other file for reading causes some
> issues like this. It would be really helpful if someone help me clear
> this issue. The getattr() calls are actually called upon the original
> file (doing an os.lstat()). Does this make any effect on the 'less'
> output ?

I'm afraid we are getting beyond my level of fuse-foo here.  You'd
probably be better off finding a fuse or even fuse-python mailing list
and trying there.

If it were me, I'd start logging everything I could (take a look at
Python's 'logging' module to help you make that easy), and twidling
things.  What happens if you change what gets returned to lstat?
What happens for various sizes and contents of the '0' file?  What
happens if you use 'cat -v' or hexdump instead of less to read the file?
Run experiments until you gather enough clues to make a guess as to what
is going on, then test your theory.  Repeat until success :)

--
R. David Murray           http://www.bitdance.com

On Mar 24, 4:45=A0pm, "R. David Murray" <rdmur...@bitdance.com> wrote:
> Sreejith K <sreejith...@gmail.com> wrote:
> > On Mar 24, 2:12=A0pm, Ant <ant...@gmail.com> wrote:
> > > On Mar 24, 7:59=A0am, Sreejith K <sreejith...@gmail.com> wrote:
> > > ...
>
> > > > data is the whole file, but 'less' gives only the two lines...
>
> > > From this statement (that you are using less), it appears that you ar=
e
> > > redirecting sys.stdout to a file or similar - if that is the case, yo=
u
> > > may need to flush or close the output file before it picks up any
> > > changes.
>
> It's not a redirect to a file. =A0Fuse calls the 'read' function on the
> class, the read function does a 'return' of the data, and fuse passes
> the data up through the OS layer to be the result of the 'read' call
> made by less.
>
> If you don't know what a fuse file system is, this all gets very confusin=
g :)
>
> > Yes, I did try a flush() and close on the file which is read, but the
> > result is the same. I think when a read comes in fuse redirecting the
> > actual path (original file) to some other file for reading causes some
> > issues like this. It would be really helpful if someone help me clear
> > this issue. The getattr() calls are actually called upon the original
> > file (doing an os.lstat()). Does this make any effect on the 'less'
> > output ?
>
> I'm afraid we are getting beyond my level of fuse-foo here. =A0You'd
> probably be better off finding a fuse or even fuse-python mailing list
> and trying there.
>
> If it were me, I'd start logging everything I could (take a look at
> Python's 'logging' module to help you make that easy), and twidling
> things. =A0What happens if you change what gets returned to lstat?
> What happens for various sizes and contents of the '0' file? =A0What
> happens if you use 'cat -v' or hexdump instead of less to read the file?
> Run experiments until you gather enough clues to make a guess as to what
> is going on, then test your theory. =A0Repeat until success :)
>
> --
> R. David Murray =A0 =A0 =A0 =A0 =A0http://www.bitdance.com

Thanks Murray, for your valuable opinions and suggestions. I'll try to
log everything from now on and find out what happens inside. :)
Surely I'll let you guys know what happened..

> It's not a redirect to a file. =A0Fuse calls the 'read' function on the
> class, the read function does a 'return' of the data, and fuse passes
> the data up through the OS layer to be the result of the 'read' call
> made by less.

By redirection I meant reading the snapshot file instead of the
original file :). Sorry for making confusions....