Hi everyone,

Any work-around for opening files larger than 2GB using fopen() in stdio.h?
fopen() will report "file size too large" when I open any file larger than
2^31 - 1 (around 2GB).

Tommy wrote :

> Any work-around for opening files larger than 2GB using fopen() in stdio.h?
> fopen() will report "file size too large" when I open any file larger than
> 2^31 - 1 (around 2GB).

you can use read(), write(), lseek()

or mmap() to access smaller parts of the file.

-- 
DINH V. Hoa,

"�a doit �tre une racaille pour �tre aussi con" -- Cent-Quarante-Six

Tommy <tommy@nospam.com> writes:

> Any work-around for opening files larger than 2GB using fopen() in stdio.h?
> fopen() will report "file size too large" when I open any file larger than
> 2^31 - 1 (around 2GB).

Use fopen64 or #define _FILE_OFFSET_BITS 64.  I'm not sure how
portable the last one is.

-- 
M�ns Rullg�rd
mru@kth.se

Hi,

> Use fopen64 or #define _FILE_OFFSET_BITS 64.  I'm not sure how
> portable the last one is.

fopen64() works.  However, GNU gcc save me warnings as if the fopen64() is
not defined.  How to work this out beside turning off warning?

btw, how large can fopen64(), fseek64(), *64() can handle?  The man page
just say "large file."



M�ns Rullg�rd wrote:

> Tommy <tommy@nospam.com> writes:
> 
>> Any work-around for opening files larger than 2GB using fopen() in
>> stdio.h? fopen() will report "file size too large" when I open any file
>> larger than 2^31 - 1 (around 2GB).
> 
> Use fopen64 or #define _FILE_OFFSET_BITS 64.  I'm not sure how
> portable the last one is.
> 

mru@kth.se wrote :

> Use fopen64 or #define _FILE_OFFSET_BITS 64.  I'm not sure how
> portable the last one is.

are you sure fopen64 is portable ?
I cannot see that call in POSIX, nor in man pages of Solaris, FreeBSD 
(nor linux in fact).

-- 
DINH V. Hoa,

"�a doit �tre une racaille pour �tre aussi con" -- Cent-Quarante-Six

http://www.mkssoftware.com/docs/man3/fopen.3.asp
> are you sure fopen64 is portable ?
> I cannot see that call in POSIX, nor in man pages of Solaris, FreeBSD
> (nor linux in fact).

It works in linux.  However, there is no man page for it and gcc complains
for not knowing fopen64().  If you compile anyway, the function will
resolve at runtime.  I couldn't get fopen64() and fseeko64() to compile on
windows however (it couldn't resolve at runtime).

using #define _FILE_OFFSET_BITS 64 didn't solve the "file too large" problem
of fopen().  I tried it in Linux.  The man page of fseeko() in linux did
mention about using this define statement for that purpose but it didn't
work w/ fopen().


DINH Viet Hoa wrote:

> mru@kth.se wrote :
> 
>> Use fopen64 or #define _FILE_OFFSET_BITS 64.  I'm not sure how
>> portable the last one is.
> 
> are you sure fopen64 is portable ?
> I cannot see that call in POSIX, nor in man pages of Solaris, FreeBSD
> (nor linux in fact).
> 

Tommy wrote :

> It works in linux.  However, there is no man page for it and gcc complains
> for not knowing fopen64().  If you compile anyway, the function will
> resolve at runtime.  I couldn't get fopen64() and fseeko64() to compile on
> windows however (it couldn't resolve at runtime).

if you define _GNU_SOURCE (*sigh*) before including stdio.h, I think the 
compiler will not complain more.

-- 
DINH V. Hoa,

"�a doit �tre une racaille pour �tre aussi con" -- Cent-Quarante-Six

DINH Viet Hoa <dinh.viet.hoa@free.fr> writes:

>> It works in linux.  However, there is no man page for it and gcc complains
>> for not knowing fopen64().  If you compile anyway, the function will
>> resolve at runtime.  I couldn't get fopen64() and fseeko64() to compile on
>> windows however (it couldn't resolve at runtime).
>
> if you define _GNU_SOURCE (*sigh*) before including stdio.h, I think the 
> compiler will not complain more.

#define _LARGE_FILE_SOURCE is the proper, and possibly more portable, way.

-- 
M�ns Rullg�rd
mru@kth.se

Tommy <tommy@nospam.com> writes:

> It works in linux.  However, there is no man page for it and gcc complains
> for not knowing fopen64().  If you compile anyway, the function will
> resolve at runtime.  I couldn't get fopen64() and fseeko64() to compile on
> windows however (it couldn't resolve at runtime).

The glibc info manual mentions it.

> using #define _FILE_OFFSET_BITS 64 didn't solve the "file too large" problem
> of fopen().  I tried it in Linux.  The man page of fseeko() in linux did
> mention about using this define statement for that purpose but it didn't
> work w/ fopen().

You have to do that define before you include any system header
files.  You might get away with defining it before #include <stdio.h>,
but some other header could possibly include stdio.h, so it's safer to
define it before any #includes.

-- 
M�ns Rullg�rd
mru@kth.se

Thank you all for your help.  Here is the summary on my Linux system:

1.  Using #define _FILE_OFFSET_BITS 64 before any other #define did allows
fopen() to open larger than 2GB file.
2.  Using #define _FILE_OFFSET_BITS 64 did NOT allow fseek() OR fseeko() to
seek more than 2GB ahead, however.
3.  Using #define _GNU_SOURCE did allow fseeko() to seek more than 2GB ahead
AND get rid of gcc warning message about fseeko() not defined.
4.  Using #define _GNU_SOURCE did NOT allow fseek() to seek more than 2GB
ahead, however.
5.  Using #define _LARGE_FILE_SOURCE seems to have no effect on my system.



DINH Viet Hoa wrote:

> Tommy wrote :
> 
>> It works in linux.  However, there is no man page for it and gcc
>> complains
>> for not knowing fopen64().  If you compile anyway, the function will
>> resolve at runtime.  I couldn't get fopen64() and fseeko64() to compile
>> on windows however (it couldn't resolve at runtime).
> 
> if you define _GNU_SOURCE (*sigh*) before including stdio.h, I think the
> compiler will not complain more.
> 

On Fri, 21 Nov 2003 11:07:13 GMT, Tommy <tommy@nospam.com> wrote:

>Hi,
>
>> Use fopen64 or #define _FILE_OFFSET_BITS 64.  I'm not sure how
>> portable the last one is.
>
>fopen64() works.  However, GNU gcc save me warnings as if the fopen64() is
>not defined.  How to work this out beside turning off warning?

grep your include directories to find which one has the prototype.

-- 
Al Balmer
Balmer Consulting
removebalmerconsultingthis@att.net

On Fri, 21 Nov 2003, Tommy wrote:

> Any work-around for opening files larger than 2GB using fopen() in stdio.h?
> fopen() will report "file size too large" when I open any file larger than
> 2^31 - 1 (around 2GB).

Which OS?  Whatever, you must build your application so that
it is large file aware; i.e. pass -D_FILE_OFFSET_BITS=64 to
the compiler.

-- 
Rich Teer, SCNA, SCSA

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-online.net

Rich Teer wrote:
> On Fri, 21 Nov 2003, Tommy wrote:
> 
> 
>>Any work-around for opening files larger than 2GB using fopen() in stdio.h?
>>fopen() will report "file size too large" when I open any file larger than
>>2^31 - 1 (around 2GB).
> 
> 
> Which OS?  Whatever, you must build your application so that
> it is large file aware; i.e. pass -D_FILE_OFFSET_BITS=64 to
> the compiler.

For the record, this is only true when the OS is 32-bit (or a hybrid 
like Solaris). On a fully 64-bit OS the -D_FILE_OFFSET_BITS=64 is an 
anachronism, though harmless.

MB

On Fri, 21 Nov 2003, Mohun Biswas wrote:

> For the record, this is only true when the OS is 32-bit (or a hybrid
> like Solaris). On a fully 64-bit OS the -D_FILE_OFFSET_BITS=64 is an
> anachronism, though harmless.

Yes, that's true.  I meant to add "Or build your app as a 64-bit one",
but forgot to before I hit send.

-- 
Rich Teer, SCNA, SCSA

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-online.net