Adding Support for Arbitrary File Sizes to the Single UNIX Specification

Last Update: 20Mar96


Table of Contents

Adding Support for Arbitrary File Sizes to the Single UNIX Specification
1.0 Overview
1.1 The Large File Problem
1.2 Requirements
1.3 Importance
1.4 Concepts
1.5 Changes and Additions
1.6 Conformance
2.0 Changes to the Single UNIX Specification
2.1 Changes to CAE Specification System Interface Definitions, Issue 4, Version 2
2.2 Changes to CAE Specification System Interfaces and Headers, Issue 4, Version 2
2.2.1 Changes to System Interfaces
2.2.2 Changes to Headers
2.3 Changes to CAE Specification Commands and Utilities, Issue 4, Version 2
3.0 Transitional Extensions to the Single UNIX Specification
3.1 Transitional Extensions to CAE Specification System Interfaces and Headers, Issue 4, Version 2
3.1.1 Transitional Extensions to System Interfaces
3.1.2 Transitional Extensions to Headers
3.2 Transitional Extensions to the mount Utility
3.3 Accessing the Extensions to the SUS

Appendix A: Rationale and Notes
A.1 Overview
A.1.1 Guiding Principles
A.1.2 Concepts
A.2 Changes to the Single UNIX Specification
A.2.1 Changes to CAE Specification System Interfaces and Headers, Issue 4, Version 2
A.2.1.1 Changes to System Interfaces
A.2.2 Changes to CAE Specification Commands and Utilities, Issue 4, Version 2
A.3 Transitional Extensions to the Single UNIX Specification
A.3.1 Transitional Extensions to CAE Specification System Interfaces and Headers, Issue 4, Version 2
A.3.1.1 Transitional Extensions to System Interfaces
A.3.1.2 Transitional Extensions to Headers
A.3.2 Accessing the Transitional Extensions to the SUS

Acknowledgements

Revision Information
23Feb96 Version 1.1
24Feb96 Version 1.2
01Mar96 Version 1.3
05Mar96 Version 1.4
20Mar96 Version 1.5

Acknowledgements

Even with the rise of 64-bit systems, the 32-bit operating system will be with us for a while yet. However, the need for interoperability with 64-bit systems, large applications, large databases, and cheap disks has created a market imperative for the UNIX industry: support large files on 32-bit systems. Most current UNIX systems support file sizes of at most 2^31-1 bytes. This is not enough for today's applications, which include files containing videos, sounds, images, and large databases. Today's 32-bit systems are quite capable of handling the computational needs of these applications, but they need to be able to support maximum file sizes that are many orders of magnitude larger.

This support must be compatible with the existing Single UNIX Specification, and provide a path to conformance with following versions. It must allow system vendors a cost effective approach to adding these features to their existing products, and provide system vendors, software vendors, and users with a clear path for future products. The independent software vendors (ISVs) listed below gathered in a set of meetings with the UNIX systems vendors to develop a common set of APIs and modifications to the Single UNIX Specification to allow support for large files. We called these meetings the Large File Summit. For details of the meetings, how the proposals were developed, and the ISV requirements document, see http://www.sas.com/standards/large.file.

This work is being sent to the X/Open Base System Working Group so they can consider the changes that are suggested for the next generation of the Single UNIX Specification.

The individuals who participated in the Large File Summit meetings and on-line discussions were:

Amdahl Corp.:  Dennis Chapman, John Haines
Convex Computer Corp.:  Mike Carl, Peter Poorman, Tom White
Cray Research, Inc.:  Rick Matthews
Data General Corp.:  Dean Herington
Digital Equipment Corp.:  Fred Glover, Ray Lanza, Peter Smith
Fujitsu:  Chris Seabrook
HAL Computer Systems, Inc.:  Prashant Dholakia, Howard Gayle,
     David H. Yamada
Hewlett-Packard Co.:  Larry Dwyer, Hal Prince
IBM Corp.:  Bill Baker, Mark Brown
MacNeal-Schwendler Corp.:  David Lombard
NCR:  Kevin Brasche, Shawn Shealy
NEC Systems Laboratory, Inc.:  Jeff Forys
Novell:  Bill Cox, John Kiger, Seth Rosenthal
NOVON Research Inc.:  Brian Boyle
Oracle:  Mark Johnson
Programmed Logic Corp.:  Tim Williams, Steve Rago
Pyramid Technology Corporation:  Ralph Campbell, Henry Robinson
SAS Institute Inc.:  Mark Cates, Leigh Ihnen, Tom Truscott,
     Kelly Wyatt
Sequent Computer Systems:  Gerrit Huizenga, Mike Spitzer
Siemens Nixdorf Inc.:  Ralf Nolting, Klaus Thon
Silicon Graphics:  Steve Cobb, Adam Sweeney
Stratus Computer Inc.:  Tony Luck 
Sun Microsystems, Inc.:  Steve Chessin
SunSoft Inc.:  Karen Barnes, Don Cragun, Karl Danz, Andy Roach,
     Glenn Skinner, Peter Van der Linden,
     Srinivasan Viswanathan
Sybase Inc.:  Marc Sugiyama
Syncsort Inc.:   Asokan
Tandem Computers:  David M. VomLehn
The Santa Cruz Operation, Inc.:  John Farley, Kurt Gollhardt,
     Art Herzog, Danielle Lahmani, Wen-Ling Lu, Dave Prosser
Unisoft:  Guy Hadland
Unisys Corp.:  Steve Beck, Bruce Jones, Scott Lurndal,
     Jim Soddy
UTG Inc.:  Michael Dortch, Mark Hatch, Larry Lytle
Veritas:  Craig Harmer, Michael Schmitz

Special thanks go to SAS Institute Inc., SunSoft, Silicon Graphics, and Convex Computer (now HP) for providing meeting rooms and logistics support. Hal Prince and Don Cragun provided technical guidance and kept us aware of the details. Mark Brown helped us understand how important it was to comply with existing standards. Bill Baker and Tom White worked hard typing early drafts and providing alternative ways to organize the document. Adam Sweeney and Howard Gayle kept us within reason. David VomLehn and Tom Truscott kept good notes and provided the minutes. Ray Lanza gave us rousing encouragement ("Just make everything 64 bits!!"). Mark Johnson quipped excellent summaries. Kelly Wyatt did the final edits and provided an excellent sanity check during the endgame. And special thanks go to Mark Hatch (now with Integrated Computer Solutions, Inc.) who organized the first meetings and got this effort going.

I really enjoyed participating and would like to express my gratitude to the members of the large file summit. In particular, I enjoyed participating with people who were so honestly motivated to make the right technical decisions. This was a great lesson in UNIX file system semantics and how the Open Systems Process works.

There are a couple of interesting features of this specification. First, it contains a method of supporting an industry wide transition to full 64-bit APIs. Second, it specifies a set of changes to the Single UNIX Specification that will allow unlimited file offsets. The transition includes a way to add 64-bit file indexing without breaking current compliance to standards, and allow software developers to migrate existing sources and binaries to systems that support 64-bit file indexing.

This document is the result of a collaborative process that was open to all participants. The efforts of those who participated will best be rewarded by having this work accepted and used. I believe that this specification is an example of how well the industry can work together to solve problems that affect our ability to produce products that compete in the market.

John Carl Zeigler, jcz@utg.org
VP Technology, UTG Inc.
Cary, NC


1.0 Overview

1.1 The Large File Problem

As UNIX systems have become increasingly powerful, a number of system vendors and UNIX independent software vendors have developed a requirement to access files that contain more information than can be addressed using a signed long integer. One possible solution could be to convert every program using files to a larger size for long integers, including the operating system. However, the work to do this is undesirable for many vendors. A number of major system vendors and users have been meeting at the "Large File Summit" (LFS) for over a year to develop a set of changes to the existing Single UNIX Specification (SUS) that allow both new and converted programs to address files of arbitrary sizes. This set of changes will be provided to X/Open for inclusion into the next version of the SUS. In addition, a set of transitional extensions intended to permit users to immediately implement large file support on typical 32-bit UNIX operating systems is proposed. Both the changes and transitional extensions and the rationale behind their definition is included in this document.

1.2 Requirements

The LFS has worked to develop a solution to the large file problem meeting the following requirements:
Be implementable at a reasonable cost
Several of the LFS members are leading efforts to develop and implement solutions. Results from their experiences have guided our decisions.
Protect existing programs
This proposal allows for protection of existing programs. Many of the solutions considered would have caused existing programs to fail unexpectedly and silently. This proposal has been carefully crafted to reduce this possibility.
Provide access to files much larger than 2 gigabytes on 32-bit operating systems
This is the requirement that first motivated the LFS activity. The proposed changes implement a solution that allows file size and related sizes to be uncoupled from the size of the C language data types chosen for an operating environment. As a result, systems conforming to the proposed changes to the SUS can support files of arbitrary sizes.
Be fully compliant to the SUS
Systems modified to support the proposed extensions can be configured to strictly conform to the existing SUS. These same systems will normally be configured to fully meet the proposed changes supporting arbitrary file sizes and remain compliant to the SUS with extensions. In addition, conforming systems can also support a transitional API extension designed to substantially reduce the difficulty of conversion to this proposed standard while remaining compliant to the existing SUS. This transitional interface is contained in section 3.0 Transitional Extensions to the Single UNIX Specification.
Provide an extension to the SUS
While the LFS would like to see this proposal included in the next version of the SUS, this specification provides extensions that system vendors and independent software vendors need to support this functionality in their current compliant products.

1.3 Importance

As noted earlier, several vendors have already begun or completed implementation because of substantial market pressures. Independent software vendors are already writing software dependent on large file functionality. Rapid inclusion into the SUS is necessary to avoid repeating the existing situation where over 20 different implementations of asynchronous I/O are available on various UNIX systems. The LFS has chosen design alternatives to facilitate the needed rapid process of standardization. We believe the proposed changes will substantially enhance the value of the next revision of the SUS if they are included.

1.4 Concepts

The proposed changes are motivated by a consistent implementation of a few very basic technical concepts.
Mixed sizes of off_t
During a period of transition from existing systems to systems able to support an arbitrarily large file size, most systems will need to support binaries with two or more sizes of the off_t data type (and related data types). This mixed off_t environment may occur on a system with an ABI that supports different sizes of off_t. It may occur on a system which has both a 64-bit and a 32-bit ABI. Finally, it may occur when using a distributed system where clients and servers have differing sizes of off_t. In effect, the period of transition will not end until we need 128-bit file sizes, requiring yet another transition! The proposed changes may also be used as a model for the 64 to 128-bit file size transition.
Offset maximum
Most, but unfortunately not all, of the numeric values in the SUS are protected by opaque type definitions. In theory this allows programs to use these types rather than the underlying C language data types to avoid issues like overflow. However, most existing code maps these opaque data types like off_t to long integers that can overflow for the values needed to represent the offsets possible in large files.

To protect existing binaries from arbitrarily large files, a new value (offset maximum) will be part of the open file description. An offset maximum is the largest offset that can be used as a file offset. Operations attempting to go beyond the offset maximum will return an error. The offset maximum is normally established as the size of the off_t "extended signed integral type" used by the program creating the file description.

The open() function and other interfaces establish the offset maximum for a file description, returning an error if the file size is larger than the offset maximum at the time of the call. Returning errors when the offset maximum is (or is likely to be) exceeded protects existing binaries effectively.

EOVERFLOW
In a system with binaries compiled to support different sizes of off_t, operations such as read() or write() can attempt to reach parts of a large file beyond the range of an off_t or other limit. The existing SUS does not define an error for this case. EOVERFLOW is an existing error type that must be added to a number of system interfaces to communicate the new error condition to applications.
Development models
In addition to supporting environments requiring mixed sizes of off_t, the LFS also considered the development model. To maintain older programs that have not been converted to support arbitrary file sizes, it is necessary to specify the size of off_t and related data types. Two compilation models and the means to control them are specified in section 3.3 Accessing the Extensions to the SUS. A new set of transitional extensions will probably be needed when the next jump to larger file sizes occurs. The changes specified for the SUS, however, are size neutral.
Selectable off_t
In this model, the size of off_t is specified at compile time, and the appropriate set of libraries, headers and data types is chosen during the compilation and linking process. All existing binaries default to an off_t the size of a long integer.
Explicit off_t
In this model, the size of off_t is specified during application design. The system interface specified explicitly uses an off_t of a particular length. On a 32-bit system, for example, use of open() implies an off_t of 32 bits and use of open64() implies an off64_t of 64 bits. While the model is very useful for supporting incremental conversions and writing system software, it is not directly supported in the SUS. A proposed set of transitional extensions is described in section 3.0 Transitional Extensions to the Single UNIX Specification. These transitional interfaces support only the 32-bit to 64-bit file offset transition.

1.5 Changes and Additions

The requirements and concepts defined above have been consistently and completely applied to the SUS to generate the changes and additions specified in sections 2.0 Changes to the Single UNIX Specification and 3.0 Transitional Extensions to the Single UNIX Specification. The changes are classified as:
Changes to System Interface Definitions
The terms extended signed integral type, extended unsigned integral type, offset maximum and saved resource limits have been defined.
Changes to System Interfaces and Headers
EOVERFLOW, EFBIG and EINVAL are added or updated wherever needed.

The open() and fcntl() functions have been changed to support the offset maximum.

The fseeko() and ftello() functions have been added because the existing fseek() and ftell() do not use the required opaque types.

Data types, declarations and symbolic constants were added to or changed in headers.

Changes to Commands and Utilities
Utilities needed to establish a minimally complete system that can support large files which require conversion are defined. A complete conversion is both expensive and unnecessary for effective use of large files.
Transitional Extensions
The proposed transitional extensions including interfaces, macros and data types have been defined.

1.6 Conformance

A conforming implementation will supply all the interfaces that are specified in 2.0 Changes to the Single UNIX Specification (except that implementations need not provide the asynchronous I/O interfaces: aio_read(), aio_write(), and lio_listio()) and will define _LFS_LARGEFILE to be 1 (see 3.1.2.12 <unistd.h>).

A conforming implementation that provides asynchronous I/O interfaces and the extensions to them specified in 2.0 Changes to the Single UNIX Specification will define _LFS_ASYNCHRONOUS_IO to be 1 (see 3.1.2.12 <unistd.h>).

A conforming implementation that provides the explicit 64-bit interfaces will provide at least those interfaces specified in 3.1.1.1.3 Other Interfaces, 3.1.1.2 fcntl(), 3.1.1.3 open(), and 3.1.2 Transitional Extensions to Headers (except that changes specified in 3.1.2.2 <aio.h> and 3.1.2.6 <stdio.h> need not be supported) and will define _LFS64_LARGEFILE to be 1 (see 3.1.2.12 <unistd.h>).

A conforming implementation that defines _LFS64_LARGEFILE to be 1 and provides the explicit 64-bit interfaces for asynchronous I/O specified in 3.1.1.1.1 Asynchronous I/O Interfaces will define _LFS64_ASYNCHRONOUS_IO to be 1 (see 3.1.2.12 <unistd.h>).

A conforming implementation that defines _LFS64_LARGEFILE to be 1 and provides the explicit 64-bit STDIO interfaces specified in 3.1.1.1.2 STDIO Interfaces and 3.1.2.6 <stdio.h> will define _LFS64_STDIO to be 1 (see 3.1.2.12 <unistd.h>).

2.0 Changes to the Single UNIX Specification

2.1 Changes to CAE Specification System Interface Definitions, Issue 4, Version 2

The following definitions will be added to System Interface Definitions, Chapter 2, Glossary:
extended signed integral type
a signed integral type or an implementation-specific type with similar properties.
extended unsigned integral type
an unsigned integral type or an implementation-specific type with similar properties.
offset maximum
an attribute of an open file description representing the largest value that can be used as a file offset.
saved resource limits
an attribute of a process that provides some flexibility in the handling of unrepresentable resource limits, as described in the exec family of functions and setrlimit().

(Note the attribute "resource limits" as used in the SUS is not defined.)

2.2 Changes to CAE Specification System Interfaces and Headers, Issue 4, Version 2

2.2.1 Changes to System Interfaces

The following changes will be made to System Interfaces and Headers, Chapter 3, System Interfaces. The Asynchronous I/O interfaces (aio_read(), aio_write() and lio_listio()) should be included when POSIX.1b is added in a future revision to the SUS.

2.2.1.1 aio_read()

DESCRIPTION
For regular files, no data transfer will occur past the offset maximum established in the open file description associated with aiocbp->aio_fildes.
ERRORS
The following is an additional condition which may be detected synchronously or asynchronously:
[EOVERFLOW]
The file is a regular file, aiocbp->aio_nbytes is greater than 0 and the starting offset in aiocbp->aio_offset is before the end-of-file and is at or beyond the offset maximum in the open file description associated with aiocbp->aio_fildes.
Note: This is a new error condition.

2.2.1.2 aio_write()

DESCRIPTION
For regular files, no data transfer will occur past the offset maximum established in the open file description associated with aiocbp->aio_fildes.
ERRORS
The following is an additional condition which may be detected synchronously or asynchronously:
[EFBIG]
The file is a regular file, aiocbp->aio_nbytes is greater than 0 and the starting offset in aiocbp->aio_offset is at or beyond the offset maximum in the open file description associated with aiocbp->aio_fildes.
Note: This is an additional EFBIG error condition.

2.2.1.3 exec

DESCRIPTION
The saved resource limits in the new process image are set to be a copy of the process's corresponding hard and soft resource limits.

2.2.1.4 fclose(), fflush(), fputwc(), fputws(), fseek(), putwc(), putwchar()

ERRORS
These functions will fail if:
[EFBIG]
The file is a regular file and an attempt was made to write at or beyond the offset maximum associated with the corresponding stream.
Note: This is an additional EFBIG error condition.

2.2.1.5 fcntl()

DESCRIPTION
An unlock (F_UNLCK) request in which l_len is non-zero and the offset of the last byte of the requested segment is the maximum value for an object of type off_t, when the process has an existing lock in which l_len is 0 and which includes the last byte of the requested segment, will be treated as a request to unlock from the start of the requested segment with an l_len equal to 0. Otherwise an unlock (F_UNLCK) request will attempt to unlock only the requested segment.
ERRORS
The fcntl() function will fail if:
[EOVERFLOW]
One of the values to be returned cannot be represented correctly.
[EOVERFLOW]
The cmd argument is F_GETLK, F_SETLK or F_SETLKW and the smallest or, if l_len is non-zero, the largest, offset of any byte in the requested segment cannot be represented correctly in an object of type off_t.
Note: These are new error conditions.

2.2.1.6 fdopen()

DESCRIPTION
The fdopen() function will preserve the offset maximum previously set for the open file description corresponding to fildes.

2.2.1.7 fgetc(), fgets(), fgetwc(), fgetws(), fread(), fscanf(), getc(), getchar(), gets(), getw(), getwc(), getwchar(), scanf()

ERRORS
These functions will fail if data needs to be read and:
[EOVERFLOW]
The file is a regular file and an attempt was made to read at or beyond the offset maximum associated with the corresponding stream.
Note: This is a new error condition.

2.2.1.8 fgetpos()

ERRORS
The fgetpos() function will fail if:
[EOVERFLOW]
The current value of the file position cannot be represented correctly in an object of type fpos_t.
Note: This is a new error condition.

2.2.1.9 fopen(), freopen(), tmpfile()

DESCRIPTION
The largest value that can be represented correctly in an object of type off_t will be established as the offset maximum in the open file description.
ERRORS
The fopen() and freopen() functions will fail if:
[EOVERFLOW]
The named file is a regular file and the size of the file cannot be represented correctly in an object of type off_t.
Note: This is a new error condition.

2.2.1.10 fpathconf() and pathconf()

DESCRIPTION
  Variable          Value of name          Notes
  FILESIZEBITS      _PC_FILESIZEBITS       3,4

2.2.1.11 fprintf(), fputc(), fputs(), fwrite(), printf(), putc(), putchar(), puts(), putw(), vfprintf(), vprintf()

ERRORS
These functions will fail if either the stream is unbuffered or the stream's buffer needed to be flushed and:
[EFBIG]
The file is a regular file and an attempt was made to write at or beyond the offset maximum.
Note: This is an additional EFBIG error condition.

2.2.1.12 fseek()

ERRORS
The fseek() function will fail if:
[EOVERFLOW]
The resulting file offset would be a value which cannot be represented correctly in an object of type long.
Note: This is a new error condition.

2.2.1.13 fseeko()

DESCRIPTION
The fseeko() function is identical to the modified fseek() except that the offset argument is of type off_t and the EOVERFLOW error is changed as follows:
ERRORS
[EOVERFLOW]
The resulting file offset would be a value which cannot be represented correctly in an object of type off_t.
Note: This is a new function.

2.2.1.14 fstat(), lstat() and stat()

ERRORS
These functions will fail if:
[EOVERFLOW]
The file size in bytes or the number of blocks allocated to the file or the file serial number cannot be represented correctly in the structure pointed to by buf.
Note: This is an additional EOVERFLOW error condition.

2.2.1.15 fstatvfs() and statvfs()

ERRORS
These functions will fail if:
[EOVERFLOW]
One of the values to be returned cannot be represented correctly in the structure pointed to by buf.
Note: This is a new error condition.

2.2.1.16 ftell()

ERRORS
The ftell() function will fail if:
[EOVERFLOW]
The current file offset cannot be represented correctly in an object of type long.
Note: This is a new error condition.

2.2.1.17 ftello()

DESCRIPTION
The ftello() function is identical to the modified ftell() except that the return value is of type off_t and the EOVERFLOW error is changed as follows:
ERRORS
[EOVERFLOW]
The current file offset cannot be represented correctly in an object of type off_t.
Note: This is a new function.

2.2.1.18 ftruncate()

ERRORS
The ftruncate() function will fail if:
[EFBIG]
The file is a regular file and length is greater than the offset maximum established in the open file description associated with fildes.
Note: This is an additional EFBIG error condition.

2.2.1.19 getrlimit() and setrlimit()

DESCRIPTION
When using the getrlimit() function, if a resource limit can be represented correctly in an object of type rlim_t then its representation is returned; otherwise if the value of the resource limit is equal to that of the corresponding saved hard limit the value returned is RLIM_SAVED_MAX; otherwise the value returned is RLIM_SAVED_CUR.

When using the setrlimit() function, if the requested new limit is RLIM_INFINITY the new limit will be "no limit"; otherwise if the requested new limit is RLIM_SAVED_MAX the new limit will be the corresponding saved hard limit; otherwise if the requested new limit is RLIM_SAVED_CUR the new limit will be the corresponding saved soft limit; otherwise the new limit will be the requested value. In addition, if the corresponding saved limit can be represented correctly in an object of type rlim_t then it will be overwritten with the new limit.

The result of setting a limit to RLIM_SAVED_MAX or RLIM_SAVED_CUR is unspecified unless a previous call to getrlimit() returned that value as the soft or hard limit for the corresponding resource limit.

The determination of whether a limit can be correctly represented in an object of type rlim_t is implementation-dependent. For example, some implementations permit a limit whose value is greater than RLIM_INFINITY and others do not.

The exec family of functions also cause resource limits to be saved. (See 2.2.1.3 exec).

2.2.1.20 lio_listio()

DESCRIPTION
For regular files, no data transfer will occur past the offset maximum established in the open file description associated with aiocbp->aio_fildes.
ERRORS
The following are additional error codes which may be set for each aiocb control block:
[EOVERFLOW]
The aiocbp->aio_lio_opcode is LIO_READ, the file is a regular file, aiocbp->aio_nbytes is greater than 0, and the aiocbp->aio_offset is before the end-of-file and is greater than or equal to the offset maximum in the open file description associated with aiocbp->aio_fildes.
[EFBIG]
The aiocbp->aio_lio_opcode is LIO_WRITE, the file is a regular file, aiocbp->aio_nbytes is greater than 0, and the aiocbp->aio_offset is greater than or equal to the offset maximum in the open file description associated with aiocbp->aio_fildes.
Note: These are additional EFBIG and EOVERFLOW error conditions.

2.2.1.21 lockf()

DESCRIPTION
An F_ULOCK request in which size is non-zero and the offset of the last byte of the requested section is the maximum value for an object of type off_t, when the process has an existing lock in which size is 0 and which includes the last byte of the requested section, will be treated as a request to unlock from the start of the requested section with a size equal to 0. Otherwise an F_ULOCK request will attempt to unlock only the requested section.

ERRORS
The lockf() function will fail if:
[EINVAL]
The function argument is not one of F_LOCK, F_TLOCK, F_TEST or F_ULOCK; or size plus the current file offset is less than 0.
[EOVERFLOW]
The offset of the first, or if size is not 0 then the last, byte in the requested section cannot be represented correctly in an object of type off_t.
Note: This is a clarification of the EINVAL error condition.
Note: EOVERFLOW is a new error condition.

2.2.1.22 lseek()

ERRORS
The lseek() function will fail if:
[EOVERFLOW]
The resulting file offset would be a value which cannot be represented correctly in an object of type off_t.
Note: This is a new error condition.

2.2.1.23 mmap()

ERRORS
The mmap() function will fail if:
[EOVERFLOW]
The file is a regular file and the value of off plus len exceeds the offset maximum established in the open file description associated with fildes.
Note: This is a new error condition.

2.2.1.24 open()

DESCRIPTION
The largest value that can be represented correctly in an object of type off_t will be established as the offset maximum in the open file description.

ERRORS
The open() function will fail if:
[EOVERFLOW]
The named file is a regular file and the size of the file cannot be represented correctly in an object of type off_t.
Note: This is a new error condition.

2.2.1.25 read() and readv()

DESCRIPTION
For regular files, no data transfer will occur past the offset maximum established in the open file description associated with fildes.

ERRORS
The read() and readv() functions will fail if:
[EOVERFLOW]
The file is a regular file, nbyte is greater than 0, the starting position is before the end-of-file and the starting position is greater than or equal to the offset maximum established in the open file description associated with fildes.
Note: This is a new error condition.

2.2.1.26 readdir()

ERRORS
The readdir() function will fail if:
[EOVERFLOW]
One of the values in the structure to be returned cannot be represented correctly.
Note: This is a new error condition.

2.2.1.27 write() and writev()

DESCRIPTION
For regular files, no data transfer will occur past the offset maximum established in the open file description associated with fildes.
ERRORS
These functions will fail if:
[EFBIG]
The file is a regular file, nbyte is greater than 0 and the starting position is greater than or equal to the offset maximum established in the open file description associated with fildes.
Note: This is an additional EFBIG error condition.

2.2.2 Changes to Headers

The following changes will be made to System Interfaces and Headers, Chapter 4, Headers.

2.2.2.1 <limits.h>

The following symbolic constant is defined as a Pathname Variable Value:
Name             Description                Acceptable Value
FILESIZEBITS     Minimum number of bits             *
                 needed to represent,
                 as a signed integer
                 value, the maximum size
                 of a regular file
                 allowed in the
                 specified directory.

2.2.2.2 <stdio.h>

The following are declared as functions and may also be defined as macros:
int         fseeko(FILE *stream, off_t offset, int whence);
off_t       ftello(FILE *stream);
The type off_t is defined through typedef as described in <sys/types.h>.

2.2.2.3 <sys/resource.h>

The following symbolic constants are defined:
RLIM_SAVED_MAX     A value of type rlim_t indicating an
                   unrepresentable saved hard limit.
RLIM_SAVED_CUR     A value of type rlim_t indicating an
                   unrepresentable saved soft limit.
On implementations where all resource limits are representable in an object of type rlim_t, RLIM_SAVED_MAX and RLIM_SAVED_CUR need not be distinct from RLIM_INFINITY.

2.2.2.4 <sys/stat.h>

The type of st_blocks in the stat structure will be changed to:
blkcnt_t    st_blocks   number of blocks allocated for this
                        object.

2.2.2.5 <sys/statvfs.h>

The types of the fields below in the statvfs structure will be changed to:
fsblkcnt_t  f_blocks    total number of blocks in the file
                        system in units of f_frsize.
fsblkcnt_t  f_bfree     total number of free blocks.
fsblkcnt_t  f_bavail    number of free blocks available to
                        non-privileged process.
fsfilcnt_t  f_files     total number of file serial numbers.
fsfilcnt_t  f_ffree     total number of free file serial
                        numbers.
fsfilcnt_t  f_favail    number of free file serial numbers
                        available to non-privileged process.

2.2.2.6 <sys/types.h>

The following data types will be defined:
blkcnt_t                Used for file block counts.
fsblkcnt_t              Used for file system block counts.
fsfilcnt_t              Used for file system file counts.

The types blkcnt_t and off_t are defined as extended signed integral types.

The types fsblkcnt_t, fsfilcnt_t, and ino_t are defined as extended unsigned integral types.

2.2.2.7 <unistd.h>

The following symbolic constant is defined for pathconf():
_PC_FILESIZEBITS

2.3 Changes to CAE Specification Commands and Utilities, Issue 4, Version 2

The following changes will be made to Commands and Utilities, Chapter 3, Utilities.

2.3.1 Considerations for Utilities in Support of Files of Arbitrary Size

Note: This is a new section and should be added to Commands and Utilities, Issue 4, Version 2, Chapter 3 after section 1.2.1, Symbolic Links.

The following utilities will support files of any size up to the maximum that can be created by the implementation. This support includes correct writing of file size related values (such as file sizes and offsets, line numbers, and block counts) and correct interpretation of command line arguments that contain such values.

basename   return non-directory portion of pathname
cat        concatenate and print files
cd         change working directory
chgrp      change file group ownership
chmod      change file modes
chown      change file ownership
cksum      write file checksums and sizes
cmp        compare two files
cp         copy files
dd         convert and copy a file
df         report free disk space
dirname    return directory portion of pathname
du         estimate file space usage
find       find files
ln         link files
ls         list directory contents
mkdir      make directories
mv         move files
pathchk    check pathnames
pwd        return working directory name
rm         remove directory entries
rmdir      remove directories
sh         shell, the standard command language interpreter
sum        print checksum and block or byte count of a file
test       evaluate expression
touch      change file access and modification times
ulimit     set or report file size limit
Exceptions to the requirement that utilities support files of any size up to the maximum are:
  1. Utilities such as tar and cpio cannot support arbitrary file sizes due to limitations imposed by fixed file formats.
  2. Uses of files as command scripts, or for configuration or control, are exempt. For example, it is not required that sh be able to read an arbitrarily large ".profile".
  3. Shell input and output redirection are exempt. For example, it is not required that the redirections sum < file or echo foo > file succeed for an arbitrarily large existing file.

2.3.2 The sh Utility

DESCRIPTION:
Pathname expansion will not fail due to the size of a file.

Shell input and output redirections will have an implementation-specific offset maximum that will be established in the open file description.

2.3.3 The pax Utility

APPLICATION USAGE
The pax utility is not able to handle arbitrary file sizes. There is currently a proposal in ballot in IEEE Project 1003.2b to address this issue.

3.0 Transitional Extensions to the Single UNIX Specification

The interfaces, macros and data types in this section are explicitly 64-bit instances of the corresponding SUS and POSIX.1b interfaces, macros and data types. The function prototype and semantics of a transitional interface will be equivalent to those of the SUS version of the call. Version test macros announcing extensions to the SUS are also defined.

The transitional extensions in this section are intended to be temporary. While an application using this specification may be using non-POSIX conforming transitional extensions to operating system functions, this does not require that system vendors break their POSIX compliance. This specification is intended to be compatible with the standards. The transitional extensions are provided so that system vendors may define a common set of large file capable extensions to their current compliant systems without violating that compliance.

3.1 Transitional Extensions to CAE Specification System Interfaces and Headers, Issue 4, Version 2

3.1.1 Transitional Extensions to System Interfaces

3.1.1.1 64-bit Versions of Interfaces

The following interfaces are explicitly 64-bit versions of the corresponding Single UNIX Specification and POSIX.1b interfaces. There is no functional difference between these and the corresponding Single UNIX Specification and POSIX.1b interfaces.

3.1.1.1.1 Asynchronous I/O Interfaces

aio_cancel64()         aio_error64()  
aio_fsync64()          aio_read64()   
aio_return64()         aio_suspend64()
aio_write64()          lio_listio64()

3.1.1.1.2 STDIO Interfaces

fgetpos64()            fopen64()      
freopen64()            fseeko64()     
fsetpos64()            ftello64()         
tmpfile64()

3.1.1.1.3 Other Interfaces

creat64()             fstat64()      
fstatvfs64()          ftruncate64()
ftw64()               getrlimit64()
lockf64()             lseek64()
lstat64()             mmap64()
nftw64()              open64()
readdir64()           setrlimit64()
stat64()              statvfs64()
truncate64()   

3.1.1.2 fcntl()

DESCRIPTION
The following additional value may be used in constructing oflag:
O_LARGEFILE
If set, the offset maximum in the open file description will be the largest value that can be represented correctly in an object of type off64_t.
The behavior of the following additional values is equivalent to the corresponding Single UNIX Specification value (FGETLK, FSETLK, FSETLKW), but they take a struct flock64 argument rather than a struct flock argument.
FGETLK64
FSETLK64
FSETLKW64

3.1.1.3 open()

DESCRIPTION
The following additional value may be used in constructing oflag:
O_LARGEFILE
If set, the offset maximum in the open file description will be the largest value that can be represented correctly in an object of type off64_t.
ERRORS
The open() function will fail if:
[EOVERFLOW]
The named file is a regular file and either O_LARGEFILE is not set and the size of the file cannot be represented correctly in an object of type off_t or O_LARGEFILE is set and the size of the file cannot be represented correctly in an object of type off64_t.
APPLICATION USAGE
Note that using open64() is equivalent to using open() with O_LARGEFILE set in oflag.

Note: For the transitional extensions these changes to open() are in place of the changes described in 2.2.1.24 open() relating to the changes to the SUS.

3.1.2 Transitional Extensions to Headers

The modifications to the headers in this section are necessary to implement the transitional extensions as described in 3.0 Transitional Extensions to the Single UNIX Specification.

3.1.2.1 64-bit Versions of Headers

In summary, the changes to the headers involve the following data types, structures and symbolic constants:

3.1.2.1.1 Data Types

blkcnt_t               fsblkcnt_t
fsfilcnt_t             fpos_t
ino_t                  off_t
rlim_t                

3.1.2.1.2 Structures

struct dirent          struct flock
struct rlimit          struct stat
struct statvfs

3.1.2.1.3 Symbolic Constants

F_GETLK                F_SETLK
F_SETLKW               RLIM_INFINITY
RLIM_SAVED_MAX         RLIM_SAVED_CUR

3.1.2.2 <aio.h>

The aiocb64 structure is defined in the same way as the aiocb structure in the POSIX.1b with the exception of the following member:
off64_t        aio_offset
The following are declared as functions and may be defined as macros.
int     aio_read64(struct aiocb64 *aiocbp);
int     aio_write64(struct aiocb64 *aiocbp);
int     lio_listio64(int mode, struct aiocb64 *const list[],
            int nent, struct sigevent *sig);
int     aio_error64(const struct aiocb64 *aiocbp);
ssize_t aio_return64(struct aiocb64 *aiocbp);
int     aio_cancel64(int fildes, struct aiocb64 *aiocbp);
int     aio_suspend64(const struct aiocb64 *const list[],
            int nent, const struct timespec *timeout);
int     aio_fsync64(int op, struct aiocb64 *aiocbp);

3.1.2.3 <dirent.h>

The dirent64 structure is defined in the same way as the dirent structure in the Single UNIX Specification with the exception of the following member:
ino64_t       d_ino     file serial number.
The following is declared as a function and may also be defined as a macro:
struct dirent64 *readdir64(DIR *dirp);

3.1.2.4 <fcntl.h>

The flock64 structure is defined in the same way as the flock structure in the Single UNIX Specification with the exception of the following members:
off64_t       l_start relative offset in bytes.
off64_t       l_len   size.
Additional values for cmd used by fcntl():
F_GETLK64     Get record locking information using struct
              flock64.
F_SETLK64     Establish a record lock using struct flock64.
F_SETLKW64    Establish a record lock, blocking, using struct
              flock64.
An additional file status flag, used by open() and fcntl(), is defined:
O_LARGEFILE     The offset maximum in the open file description
                is the largest value that can be represented
                correctly in an object of type off64_t.
The following are declared as functions and may also be defined as macros:
int     creat64(const char *path, mode_t mode);
int     open64(const char *path, int oflag, ...);

3.1.2.5 <ftw.h>

The following are declared as functions and may also be defined as macros:
int ftw64(const char *path,
    int (*fn)(const char *, const struct stat64 *, int),
    int ndirs);
int nftw64(const char *path,
    int (*fn)(const char *, const struct stat64 *, int,
               struct FTW *),
    int depth, int flags);

3.1.2.6 <stdio.h>

The following data type is defined through typedef:
fpos64_t  Type containing all information needed to specify
          uniquely every position within a file in which the
          largest offset can be represented in an object of type
          off64_t.
The following are declared as functions and may also be defined as macros:
int       fgetpos64(FILE *stream, fpos64_t *pos);
FILE     *fopen64(const char *filename, const char *mode);
FILE     *freopen64(const char *filename, const char *mode,
               FILE *stream);
int       fseeko64(FILE *stream, off64_t offset, int whence);
int       fsetpos64(FILE *stream, const fpos64_t *pos);
off64_t   ftello64(FILE *stream);
FILE     *tmpfile64(void);

3.1.2.7 <sys/mman.h>

The following is declared as a function and may also be defined as a macro:
void     *mmap64(void *addr, size_t len, int prot, int flags,
                int fd, off64_t offset);

3.1.2.8 <sys/resource.h>

The following data type is defined through typedef:
rlim64_t    type used for limit values.
The type rlim64_t must be an extended unsigned arithmetic type that can represent correctly any non-negative value of an off64_t.

The following symbolic constants are defined:

RLIM64_INFINITY    A value of type rlim64_t indicating no limit.
RLIM64_SAVED_MAX   A value of type rlim64_t indicating an
                   unrepresentable saved hard limit.
RLIM64_SAVED_CUR   A value of type rlim64_t indicating an
                   unrepresentable saved soft limit.
On implementations where all resource limits are representable in an object of type rlim64_t, RLIM64_SAVED_MAX and RLIM64_SAVED_CUR need not be distinct from RLIM64_INFINITY.

The rlimit64 structure is defined in the same way as the rlimit structure in the Single UNIX Specification with the exception of the following members:

rlim64_t  rlim_cur      the current (soft) limit.
rlim64_t  rlim_max      the hard limit.
The following are declared as functions and may also be defined as macros:
int       getrlimit64(int resource, struct rlimit64 *rlp);
int       setrlimit64(int resource, const struct rlimit64 *rlp);

3.1.2.9 <sys/stat.h>

The stat64 structure is defined in the same way as the stat structure in the Single UNIX Specification with the exception of the following members:
ino64_t     st_ino      file serial number.
off64_t     st_size     file size in bytes.
blkcnt64_t  st_blocks   number of blocks allocated for this
                        object.
The following are declared as functions and may also be defined as macros:
int         fstat64(int fildes, struct stat64 *buf);
int         lstat64(const char *, struct stat64 *buf);
int         stat64(const char *, struct stat64 *buf);

3.1.2.10 <sys/statvfs.h>

The statvfs64 structure is defined in the same way as the statvfs structure in the Single UNIX Specification with the exception of the following members:
fsblkcnt64_t  f_blocks  total number of blocks in the file
                        system in units of f_frsize.
fsblkcnt64_t  f_bfree   total number of free blocks.
fsblkcnt64_t  f_bavail  number of free blocks available to
                        non-privileged process.
fsfilcnt64_t  f_files   total number of file serial numbers.
fsfilcnt64_t  f_ffree   total number of free file serial
                        numbers.
fsfilcnt64_t  f_favail  number of free file serial numbers
                        available to non-privileged process.
The following are declared as functions and may also be defined as macros:
int         statvfs64(const char *path, struct statvfs64 *buf);
int         fstatvfs64(int fildes, struct statvfs64 *buf);

3.1.2.11 <sys/types.h>

The following data types are defined through typedef:
blkcnt64_t      Used for file block counts.
fsblkcnt64_t    Used for file system block counts.
fsfilcnt64_t    Used for file system file counts.
ino64_t         Used for file serial numbers.
off64_t         Used for file sizes.
The types blkcnt64_t and off64_t are defined as extended signed integral types.

The types fsblkcnt64_t, fsfilcnt64_t, and ino64_t are defined as extended unsigned integral types.

3.1.2.12 <unistd.h>

The following are declared as functions and may also be defined as macros:
int         lockf64(int fildes, int function, off64_t size);
off64_t     lseek64(int fildes, off64_t offset, int whence);
int         ftruncate64(int fildes, off64_t length);
int         truncate64(const char *path, off64_t length);
   
Version Test Macros:
_LFS_LARGEFILE   is defined to be 1 if the implementation
                 supports the interfaces as specified in
                 2.2.1 Changes to System Interfaces
                 except that implementations need not provide
                 the asynchronous I/O interfaces: aio_read(),
                 aio_write(), and lio_listio().
_LFS_ASYNCHRONOUS_IO
                 is defined to be 1 if the implementation
                 supports the asynchronous IO interfaces:
                 aio_read(), aio_write(), and lio_listio() as
                 specified in 2.2.1 Changes to
                 System Interfaces.
_LFS64_ASYNCHRONOUS_IO
                 is defined to be 1 if the implementation
                 supports all the transitional extensions
                 listed in 3.1.1.1.1 Asynchronous I/O Interfaces
                 and 3.1.2.2 <aio.h>.
_LFS64_LARGEFILE is defined to be 1 if the implementation
                 supports all the transitional extensions
                 listed in 3.1.1.1.3 Other Interfaces,
                 3.1.1.2 fcntl(), 3.1.1.3 open() and
                 3.1.2 Transitional Extensions to Headers,
                 except changes specified in 3.1.2.2 <aio.h>
                 and 3.1.2.6 <stdio.h> need not be supported.
_LFS64_STDIO     is defined to be 1 if the implementation
                 supports all the transitional extensions
                 listed in 3.1.1.1.2 STDIO Interfaces
                 and 3.1.2.6 <stdio.h>.
                  
                 If _LFS64_STDIO is not defined to be 1 and the
                 underlying file description associated with
                 stream has O_LARGEFILE set then the behavior
                 of the Standard I/O functions is unspecified.
                
Constants for Functions:
    _CS_LFS_CFLAGS       for confstr().
    _CS_LFS_LDFLAGS      for confstr().
    _CS_LFS_LIBS         for confstr().
    _CS_LFS_LINTFLAGS    for confstr().

    _CS_LFS64_CFLAGS     for confstr().
    _CS_LFS64_LDFLAGS    for confstr().
    _CS_LFS64_LIBS       for confstr().
    _CS_LFS64_LINTFLAGS  for confstr().

3.2 Transitional Extensions to the mount Utility

3.2.1 Optional Additional Option for the mount utility

If the -o nolargefiles option is specified and is supported by the file system, then for the duration of the mount it is guaranteed that all regular files in the file system have a file size that will fit in the smallest object of type off_t supported by the system performing the mount. The mount will fail if there are any files in the file system not meeting this criterion.

If -o largefiles is specified then there is no such guarantee.

The default behavior is implementation-dependent.

3.3 Accessing the Extensions to the SUS

3.3.1 Compilation Environment - Visibility of Additions to the API

Applications which define the macro _LARGEFILE_SOURCE to be 1 before inclusion of any header will enable at least the functionality described in 2.0 Changes to the Single UNIX Specification on implementations that support these features. Implementations that support these features will define _LFS_LARGEFILE to be 1 in <unistd.h>, as described in 3.1.2.12 <unistd.h>.

3.3.2 Compilation Environment - Visibility of Transitional API

Applications which define the macro _LARGEFILE64_SOURCE to be 1 before inclusion of any header will enable at least the fseeko(), ftello() extensions to the SUS (see 2.2.1.13 fseeko(), 2.2.1.17 ftello() and 2.2.2.2 <stdio.h>) and the transitional extensions described in 3.1 Transitional Extensions to CAE Specification System Interfaces and Headers, Issue 4, Version 2 on implementations that support these features. Implementations that support these features will define _LFS64_LARGEFILE, _LFS64_ASYNCHRONOUS_IO and _LFS64_STDIO to be 1 in <unistd.h>, as described in 3.1.2.12 <unistd.h>.

3.3.3 Mixed API and Compile Environments Within a Single Process

It is permitted to use both the Single UNIX Specification and the transitional APIs within the same executable, including within the same source file, and to use both on the same file descriptor whether in the same process or in different processes (when an open file descriptor is passed or inherited).

3.3.4 Utilities: Optional Method for Specifying the Size of an off_t

For programs to take advantage of different environments, it is necessary to compile them for each particular environment. For programs to make use of the features described in this section they must be compiled with new compiler and linker options. The getconf utility called with the new arguments can be used to generate compiler and linker options.

Example 1:

An example of compiling a program with a "large" off_t and that uses fseeko() and ftello() and uses yacc:

   c89 -D_LARGEFILE_SOURCE     -o foo      \
        $(getconf LFS_CFLAGS)  y.tab.c b.o \
        $(getconf LFS_LDFLAGS)             \
        -ly $(getconf LFS_LIBS)
Example 2:

An example of compiling a program with a "large" off_t and that does not use fseeko() and ftello() and has no application specific libraries:

   c89  $(getconf LFS_CFLAGS)  a.c         \
        $(getconf LFS_LDFLAGS)             \
        $(getconf LFS_LIBS)
Example 3:

An example of compiling a program with a "default" off_t and that uses fseeko() and ftello():

   c89 -D_LARGEFILE_SOURCE     a.c
Example 4:

An example of compiling a program using transitional versions of SUS interfaces such as lseek64() and fopen64():

   c89  -D_LARGEFILE64_SOURCE              \
        $(getconf LFS64_CFLAGS)  a.c       \
        $(getconf LFS64_LDFLAGS)           \
        $(getconf LFS64_LIBS)
Example 5:

An example of running lint on a program with a "large" off_t:

   lint -D_LARGEFILE_SOURCE                \
        $(getconf LFS_LINTFLAGS) ...       \
        $(getconf LFS_LIBS)
Example 6: An example of running lint on a program using the transitional API:
   lint -D_LARGEFILE64_SOURCE              \
        $(getconf LFS64_LINTFLAGS) ...     \
        $(getconf LFS64_LIBS)
These examples show the need for the additional variables LFS_CFLAGS, LFS_LDFLAGS, LFS_LIBS, LFS_LINTFLAGS, LFS64_CFLAGS, LFS64_LDFLAGS, LFS64_LIBS and LFS64_LINTFLAGS to be reported by getconf.

Implementations may permit the linking of object files that are compiled with differing off_t environments. For example, an object module compiled with a 32-bit off_t can be linked with an object module compiled with a 64-bit off_t. In such a case, both 32-bit off_t and 64-bit off_t API calls may be used on the same file descriptor. Implementations may instead disallow this linking.

Appendix A: Rationale and Notes

In a mixed environment the size of an off_t (and other types) might differ from program to program, and in a transitional environment (see 3.0 Transitional Extensions to the Single UNIX Specification) it might differ even from routine to routine within a single program. Each specific use of an off_t has an invariant size that is determined by the compilation environment. This is referred to below as the size which is "in use".

A.1 Overview

A.1.1 Guiding Principles

A.1.1.1 "No Lies" Rule

An error will be returned whenever a function cannot return the correct result of an operation.

Returning a "lie" to allow for common uses of a function (e.g. use of stat() to determine if a file exists) could inadvertently cause a correctly written application to operate incorrectly.

It is conceivable that returning a "lie" could keep an incorrectly written application from malfunctioning in a way that creates a serious problem, but no such applications are known to exist. (Of course it would be easy to contrive one.)

PASC Interpretation reference 1003.1-90 #38 completed by the POSIX.1 interpretations committee confirms that POSIX.1 conforming implementations are not allowed to lie to applications. This interpretation explicitly states that if the file size will not fit in an object of type off_t, fstat() must fail. In addition, PASC Interpretation reference 1003.1-90 #75 went on to clarify that EOVERFLOW would be a legal extension to report this condition.

A.1.1.2 "Open Protection" Rule

An open() will fail if the size of the (regular) file cannot be represented correctly in an object of type off_t.

The size of file on which a program is able to operate is determined by the off_t in use for the open(). The open protection rule ensures that old binaries do not operate on files that are too large to handle correctly, and prevents the binaries from generating incorrect results or corrupting the data in the file.

An argument against open protection is that requiring opens to fail will break some binaries that would have worked perfectly well otherwise. For example, a cat program does a loop of open(), read()/write() pairs, and close() for each input file. This program would unnecessarily break due to open protection. But this "Let it Run" argument is flawed in that there is no known utility which fails due to open protection but would work "perfectly well" if only we "let it run". Real versions of the cat program use fstat() to determine whether the input and output files are the same, have a -n option (count newlines) which will fail on sufficiently large files and so on.

Another argument against open protection is that it is unnecessary because an error will be returned as soon as a function cannot return the correct result of an operation ("No Lies" rule). However, most programs check for the success of the open() call, but many do not check for overflow or error after lseek() and other calls. An audit of the standard utilities uncovered numerous examples.

An argument for open protection is that it increases the likelihood of an immediate and informative error message. The error message is likely to include the name of the file that could not be opened. It is much less likely that an lseek() error message will be as immediate or as informative. The delay in, or complete lack of, reporting such errors may result in "silent failure".

Another argument for open protection is that there are numerous plausible scenarios in which this rule avoids serious harm. It prevents typical implementations of the touch utility from truncating large files to 0 length (see A.2.1.1.4 creat()). It can prevent silent failure, which has been demonstrated to occur in at least one commercial data management system. With open protection a commercial backup/restore system will report errors on files that might otherwise result in a corrupted backup tape. It prevents typical implementations of dbm/ndbm from returning incorrect results from a database whose size exceeds the off_t in use for the dbm routines.

A.1.1.3 "Read/Write Limit" Rule

For regular files, no data transfer will occur past the offset maximum established in the open file description.

There are two separate issues for this rule, which are that there is an application-dependent limit on read() and write(), and that the limit is "the offset maximum established in the open file description". The second issue is deferred to A.1.2.1 Offset Maximum. The first issue, that there be an application-dependent limit, is considered here.

There are two assertions upon which many applications rely:

  1. A file can be read until end-of-file and written until the file system is full or some other implementation limit is reached.
  2. The current file offset can be stored correctly in an object of type off_t, and any file position that can be reached with read() and write can also be reached with lseek().
In a mixed off_t environment these assertions are true only for the largest supported size of off_t. An audit of typical applications revealed that most check return codes from read() and write() in order to guard against end-of-file, full file systems, and the like, but that most do not check for overflow of file offsets or errors returned by lseek(). This suggests that it is more important to maintain the truth of the second assertion. In order to maintain the second assertion, read() and write() must not be permitted to move the file offset past the largest offset representable by the application's off_t.

The write limit avoids the unintuitive situation in which a program could create a file too large for it to open (due to open protection). This could result in a serious problem. "Can you imagine the reaction of someone who has 1.9G of data, and all of a sudden, the DBMS can no longer open the file? I wouldn't want to be working in tech support that day."

An argument for the write limit is that it keeps a program from creating a file too large for it to handle properly. An argument for the read limit is that it is a simple way to cover the hole where a file grows after it is opened.

An argument for the read/write limit rule is that generating an error at this limit provides the earliest possible warning of an incompatibility problem that could result in lost or corrupted data if the application was to continue.

An argument against the read/write limit rule is that it results in unnecessary breakage of binaries that would have worked perfectly well otherwise. This is the "Let it Run" argument, but as noted earlier few if any such programs exist.

Another argument against the read/write limit rule is that implementing it is expensive and complex. But it has already been implemented and found not to be either expensive or complex (an analysis appears in A.1.2.1 Offset Maximum).

Another argument against the read/write limit rule is that it can result in a truncated log file record (hence corrupting the log file). But this truncation and corruption can also occur due to insufficient disk space or RLIMIT_FSIZE, and indeed the standards require that this occur.

Another argument against the read/write limit rule is that instead one can use the existing file size resource limit (RLIMIT_FSIZE). But this is not a useful defense in a mixed off_t environment because it unnecessarily restricts the size of files created by programs which support a larger off_t. The practical effect will be that use of RLIMIT_FSIZE in this way will inconvenience users and they will unlimit themselves and then there will be no write limit. So this is a false, although attractive, argument.

Another argument against the read/write limit rule is that instead there can be a mount option which limits the maximum size of a file created in the file system. But regardless of other merits for such an option, it does not provide a useful defense in a mixed off_t environment because it unnecessarily restricts the size of files created by programs which support a larger off_t. The practical effect will be that the system administrator will be pressured into remounting the file system with no limit and then there will be no write limit. So this is another false, although attractive, argument.

A.1.1.4 Holes in the Protection Mechanism

The following holes in the protection mechanism are discussed in other sections of this document:

A.1.2 Concepts

A.1.2.1 Offset Maximum

The offset maximum is used to implement the read/write limit (see A.1.1.3 "Read/Write Limit" Rule). It is basically a hack to avoid the need to provide transitional versions of read()/write() and the numerous routines which call them (getchar(), putchar(), printf(), etc.). For consistency it also affects the semantics of ftruncate() and mmap().

The offset maximum is an unusual part of this specification as it is associated with the file description whereas in all other cases the limit is determined by the size of the type that is used for the call. But determining the latter for read/write would be extremely difficult in an environment in which a single process contains calls with differing sizes of off_t in use (this environment is not part of this section of the specification, but it is part of the transitional specification). In such an environment it would be necessary to determine the size of off_t for every function that might result in a read() or write(). That would include putchar(), fwrite(), fputs(), fprintf(), puts(), etc. The number of the routines that might potentially do a read() or write() is too large for such an implementation to be practical.

It is possible that while a "small" application has a file open another application with a larger off_t can extend the file beyond the size of the small application's off_t. This leads to a situation where the small application has a file descriptor which refers to a file too large for it to be able to process correctly. That is, open protection has been lost. The application will still have some protection due to "No Lies" and the "Read/Write Limit", but these are less effective protections. It is believed that this case is sufficiently unlikely that it may be safely ignored.

As an added protection, it has been suggested that all file calls should fail whenever the size of the file cannot be represented correctly in an object of type off_t. This would defend against the file growth scenario described above. But checking file size on each read/write might hurt performance in some cases and also it was not considered an important defense. It would also have the putchar(), fwrite(), etc. implementation problem.

It has been suggested that a file should not be permitted to be extended beyond the size of the smallest offset maximum in any open file description that refers to the file. It is believed that this is an unnecessary complication, cannot be enforced for some distributed file systems and applies only to a situation that it is believed may be safely ignored.

The value of the offset maximum in an open file description will not affect the semantics of operations related to other open file descriptions or of operations which create new open file descriptions, including other open file descriptions which refer to the same file.

An argument against offset maximum is that it is expensive and complex. But that is not the case. The only implementation that will matter for years is for 64-bit off_t which

Documentation would add a dozen or so lines of text, but this part of the specification does not require such documentation.

A.1.2.1.1 Offset Maximum and the 2G-1 File Size Limit

On implementations where type off_t is a 32-bit two's complement integer, the maximum value that can be correctly represented in an object of type off_t is 2^31-1 (2G-1). Because of this, the maximum file size and maximum file offset of a small file are 2G-1, but the maximum offset of any byte contained in a small file is 2G-2. An illustration of the offsets (0, 1, ...) of a file, with the bytes (b, B and L) shown as small boxes and the offset shown as "^" is:
        <- "small" -> | <- "large" >-
    ----------   -----------------------   
    | b | b | ::: | b | B | L | L | L | :::
    ^---^---^-   -^---^---^---^---^---^-   
    0   1   2     2G  2G  2G
                  -2  -1
Although an lseek() can be done to the 2G-1 offset, a read() or write() cannot be performed at that position because when B (counting number 2G, but offset 2G-1) is read or written, the resulting pointer to the next offset address and the file size itself would overflow.

A.1.2.2 Inheritance

The offset maximum will be inherited via fork(), the exec family of functions, dup(), and fcntl() called with F_DUPFD, and its value will not be altered by them. The value of the offset maximum will not affect any semantics related to inheritance.

An application can inherit, via the exec family of functions, a file descriptor that is associated with a file whose size exceeds the largest value that can be represented correctly by the off_t that is in use by the application. An example is if a shell that was compiled with a 64-bit off_t does input or output redirection of a 10 gigabyte file and then executes a program which was compiled with a 32-bit off_t. In such a case the large file unaware application will function until attempting an operation from which the results cannot be correctly returned.

Most inherited files are due to shell redirection, the other cases are rare and typically under the complete control of a single application provider. The cases that are of primary concern are:

     old_binary < large_file
and
     old_binary > large_file
In these cases a pre-existing application binary, old_binary, is given a file descriptor to a file that it would not have been able to open for itself and would be able to read and write past the limit that would have been established by the open(). The concern is that the application will do something destructive or generate incorrect results since it is not expecting a file to be so large.

In comparison, consider the following cases:

     a.out | old_binary
and
     old_binary | a.out
There is no limit to the amount of data that may be passed through a pipe. In the first case the application named a.out may push more data through the pipe than can be contained in a small file. In the second case a.out may be willing to read more data than can be contained in a small file. If a pre-existing application binary has problems with inherited file descriptors that refer to large files then it is likely to have a pre-existing problem when using a pipe for large amounts of data. While it is true that the two sets of cases are not completely equivalent, the above examples show that pre-existing binaries have had the potential to see data streams larger than the amount of data that can be contained in a small file.

Another reason it is believed that the inheritance of file descriptors does not cause problems is that the majority of existing applications do not perform seek operations on standard input or standard output.

A.1.2.3 Non-Requirements

Open protection and the read/write limit apply only to regular files, and are not specified to apply to block or character special files such as raw disk partitions.

A.1.2.4 Non-Changes

The following are to clarify, not to change, existing practice: Different files may have different maximum permitted sizes even when they are on the same system, or are on the same type of file system, or are on the same file system. The maximum permitted file sizes are independent of the offset maximum. The maximum permitted file sizes do not have specified minimum or maximum values. Attempts to grow a file via write(), writev(), or truncate() may fail even when statvfs() reports that space is available.

A.1.2.5 NFS Quality of Implementation Issue

NFS does not fall within the confines of this specification since there are no relevant NFS interfaces. However, here are some suggestions for NFS implementations.

The NFS version 2 protocol is effectively a 32-bit application since it cannot handle file sizes larger than 2^31-1 bytes. Any attempt by an NFS V2 client to access a large file (read(), write(), stat(), etc.) should be rejected by the server since the server knows the file is large and knows the application (NFS V2) is not "large file aware". This test is trivial and requires no more performance penalty than the tests for any other file system type.

The NFS version 3 protocol is "large file aware" since it can handle file sizes up to 2^63-1 bytes. An NFS V3 server would handle all requests without change, even if the request involves a large file. It is up to the NFS V3 client code to determine if the application accessing a file is "large file aware" or not. This should be handled in the standard fashion in the OS on the client side machine using the attributes returned by the NFS operation or the cached file attributes. While this does not provide perfect protection or immediate detection of files that have grown beyond 2^31-1 bytes since being opened, it is no more broken than the rest of NFS. (See below for more discussion of cached file attributes).

This does not address the issue of NFS V3 clients that are not prepared to handle "large files". If they are carefully written and obey the NFS V3 protocol they should realize that files can be larger than 2^31-1 bytes and handle this condition appropriately, probably by failing the operation (they would know this when a stat(), read(), write(), etc. operation returned a file size larger than 2^31). However, there are probably NFS V3 clients that are not carefully written. We really can't do much about that.

Cached Attributes: with the NFS V3 protocol, clients are not required to cache the file attributes, and servers are not required to return the file attributes with each operation. If the file attributes are returned with each operation, it is easy to determine if the file has grown past the large file limit. If not, the cached attributes can be consulted.

If the client does not cache attributes, then it will either have to request the attributes from the server over the wire (adversely affecting performance) or assume the file has not grown in size since it was opened. This specification pretty much requires the client code to check the file size at open.

Because of the stateless nature of NFS, it is difficult to ensure that a large-file unaware application cannot operate on a file that has grown from small to large. This is for the same reasons that NFS cannot implement standard UNIX file semantics. However, it is easy to ensure that a large-file unaware application does not grow a small file to become large (since the offset and length of each write are determined at the client, the client can fail any operation where the offset plus length exceeds the small file limit). It is also easy to insure that a large-file unaware application does not read past the small file limit.

A.2 Changes to the Single UNIX Specification

A.2.1 Changes to CAE Specification System Interfaces and Headers, Issue 4, Version 2

A.2.1.1 Changes to System Interfaces

A.2.1.1.1 Notes on Functions not Modified by this Proposal

The following functions do not require modification to meet the terms of this proposal:
aio_error(), aio_cancel(), aio_return() and aio_suspend()
No large file implications were identified for these functions.
aio_fsync()
It is possible that an aio_fsync() could try to write out file blocks that are beyond the offset maximum, just as fsync() could. There is no compelling reason for either to fail. Clearly, the original write request had to be within the offset maximum for the file description used. The aio_fsync() function will not enforce the offset maximum on the blocks which it writes out.
glob() and wordexp()
The subroutines that expand file name wild cards need to be large file capable.

A.2.1.1.2 aio_read()

The aio_read() function enforces the offset maximum rules for consistency with read() and readv().

A.2.1.1.3 aio_write()

The aio_write() function enforces the offset maximum rules for consistency with write() and writev().

A.2.1.1.4 creat()

The creat() function will fail if the named file is a regular file and the size of the file cannot be represented correctly in an object of type off_t (see 2.2.1.24 open()). This offers protection from the following coding style:
     if (stat(path, ...) < 0) {
         /* assume file does not exist, so create it */
         if ((fd = creat(path, ...)) < 0) {
            /* print out error text */
         }
     }
In this example the stat() function is being used to determine the existence of a file. But if the file size cannot be represented correctly in an object of type off_t then stat() will fail (see 2.2.1.14 fstat(), lstat() and stat()) and if creat() did not then fail it would have the unintended effect of truncating the file to 0 length. Many applications and standard utilities have code similar to this example, including typical implementations of the touch utility.

A.2.1.1.5 fcntl() and lockf()

Unlock requests are sometimes "rounded to infinity" so that a process can create a whole-file lock and then successfully issue a request to clip off the beginning of the lock without leaving behind an unrepresentable lock. This is to avoid breaking any existing 32-bit applications which might happen to do this.

Several existing implementations of fcntl() permit locking the byte whose offset is the maximum value that can be represented correctly in a object of type off_t, even though write() cannot write to that offset. This specification permits that behavior.

The fcntl() function will fail if the cmd argument is F_GETLK and the first lock which blocks the lock description has a starting offset or length which cannot be represented correctly in an object of type off_t. Information about such a lock cannot be correctly returned.

Discussion of the semantics of fcntl() locks that cross the off_t boundary resulted in six competing proposals:

  1. An unlock request fails if it would create an unrepresentable lock.
  2. If any lock request includes the byte whose offset is the maximum value that fits in an off_t, then the request is equivalent to a request where l_len is 0 and l_start refers to the first byte of the affected area.
  3. (proposal was dropped)
  4. If l_len is 0 then the lock is through and including the maximum value of off_t (and not beyond).
  5. Just no lies.
  6. If an unlock request includes the byte whose offset is the maximum value that fits in an off_t, and there is an existing lock with l_len equal to 0 which also includes that byte, then the request is equivalent to a request where l_len is 0 and l_start refers to the first byte of the affected area.

An advantage of 2, 4, and 6 is that they do not change existing behavior of a 32-bit application.

Proposals 1 and 5 can result in a new type of failure in the case where the program creates a lock with l_len equal to 0 and then clips off the beginning leaving behind an unrepresentable lock.

Proposal 4 precludes truly "whole file" locking.

Proposal 6 was adopted because as it preserves existing 32-bit behavior and is less disruptive than proposal 2 (which extends lock requests in addition to unlock requests).

The fcntl() and lockf() functions will fail if the offset of the first byte in the region, or if l_len (size) is non-zero then the offset of last byte in the region, exceeds the largest possible value in an object of type off_t. Otherwise the process could create a lock which would be "beyond" the ability of the program to represent.

A.2.1.1.6 fgetpos(), fseek(), ftell(), lseek()

These functions will fail if the resulting file offset would exceed the largest value that can be represented correctly in the related type which is in use for the call, and will set errno to EOVERFLOW (permitted by PASC Interpretation 1003.1-90 #75).

Programs typically, but incorrectly, fail to check the return value of these functions, which renders the error return less useful. On the other hand, returning an incorrect offset can result in serious malfunction as well.

An lseek() to the end of a file using

     lseek(fd, 0, SEEK_END);
is quite common. It is unfortunate that these fail on a too-large file since the return value is usually ignored. One alternative that was considered was for lseek() to move the file offset for all valid requests and then return an error if the resulting offset is too large. That is, the call would succeed for applications that do not check the return code, but also fail for applications that do check. This option was deemed too bizarre to adopt. For example, it might be difficult to implement using a remote procedure call system that was constructed to return either results or an error, but not both. In addition, the POSIX 1003.1 standard requires the file offset to remain unchanged if an error is returned by lseek(). It was felt that the open protection (see A.1.1.2 "Open Protection" Rule) and the read/write limit (see A.1.1.3 "Read/Write Limit" Rule) are more effective defenses against this problem.

Another potentially serious consequence of ignoring the return value of lseek() is that programs which extend data files by attempting to seek beyond the end-of-file and then writing may instead overwrite existing data.

For example, typical implementations of the dbm and ndbm libraries contain code such as:

     (void) lseek(db->dbm_pagf, blkno*PBLKSIZ, L_SET);
     if (write(db->dbm_pagf, pagebuf, PBLKSIZ) != PBLKSIZ)
                ... error handling ...

The problem is that the return code of lseek() is not checked and so if "blkno*PBLKSIZ" overflows the lseek() will fail (or will seek to an unintended offset) and the data will be written to an unintended offset.

A.2.1.1.7 fpathconf() and pathconf()

The reference "See Note 3,4" refers to notes in the X/Open specification for fpathconf() and pathconf(). These notes indicate that this option (_PC_FILESIZEBITS) is valid only for a directory, and the results are for files that exist or may be created in that directory.

The _PC_FILESIZEBITS option makes it possible for a process to determine how large a file can be created in a given directory. It takes into account implementation limitations in the file system (e.g. due to the size of file size and block count variables), and it takes into account long term policy limitations (e.g. due to the mount utility's -o nolargefiles option). It does not take into account dynamic restrictions such as the RLIM_FSIZE resource limit or the number of available file blocks, so the process must perform appropriate checks.

When the current directory is on a typical large file capable file system and is mounted with the -o nolargefiles option,

     pathconf(".", _PC_FILESIZEBITS);
will return 32. In general, if the maximum size file that could ever exist on the mounted file system is maxsize then the returned value is 2 plus the floor of the base 2 logarithm of maxsize.

A.2.1.1.8 fseeko() and ftello()

These functions are needed because fseek() and ftell() are limited by the long offset type required by ISO C. The fsetpos() and fgetpos() functions, although they do use an opaque offset type, are not complete replacements for fseek() and ftell() because they do not allow relative seeks or arithmetic on fpos_t values.

A.2.1.1.9 fsetpos()

Since fsetpos() sets an absolute file position, which is always legal regardless of the implementation-supported sizes of off_t, there are no new error returns or other new semantics.

A.2.1.1.10 fstatvfs() and statvfs()

These functions will fail if the total, or free, or available number of blocks or files cannot be represented correctly in the structure to be returned (f_blocks, f_bfree, f_bavail, f_files, f_ffree, f_favail).

A.2.1.1.11 ftruncate(), truncate(), unlink()

These functions are used only on pre-existing files and so do not have the potential programming hazard as does creat() (see A.2.1.1.4 creat()).

When ftruncate() is used to increase the size of a file, the semantics are similar to a write() of zeroes to the file. For consistency with write(), the ftruncate() function will fail when the request is beyond the offset maximum (even if the effect of the request would be to shorten the file).

A.2.1.1.12 ftw() and nftw()

The ftw() and nftw() functions may fail if a stat() in the underlying implementation fails with EOVERFLOW. This is unfortunate because "small" binaries using these functions cannot reasonably be used on file trees containing "large" files. Some systems have a non-standard extension to nftw() which permits it to continue when stat() fails (typical failures also include ESTALE and ELOOP).

A.2.1.1.13 getrlimit() and setrlimit()

These functions map limits that they cannot represent correctly to and from RLIM_SAVED_MAX and RLIM_SAVED_CUR. These values do not require any special handling by programs. They may be thought of as tokens that the kernel hands out to programs that can't handle the real answer, and that remind the kernel, when the tokens come back from the user, of what value is really meant.

If setrlimit() fails for any reason (for example, EPERM), the resource limits and saved resource limits remain unchanged.

This proposal does not specify any particular value for RLIM_INFINITY, RLIM_SAVED_MAX or RLIM_SAVED_CUR. Typical current implementations use the value 0x7FFFFFFF for RLIM_INFINITY, and it is recommended that RLIM_SAVED_MAX and RLIM_SAVED_CUR have similar large values.

Few, if any, programs will need to refer explicitly to RLIM_SAVED_MAX or RLIM_SAVED_CUR. Those that do should not use them in C-language switch cases since they may have the same value in some implementations (see 2.2.2.3 <sys/resource.h>).

A limit that can be represented correctly in an object of type rlim_t is either "no limit", which is represented with RLIM_INFINITY, or has a value not equal to any of RLIM_INFINITY or RLIM_SAVED_MAX or RLIM_SAVED_CUR and which can be represented correctly in an object of type rlim_t and which meets any additional implementation-specific criteria for correct representation.

A rejected alternative proposal was to map limits that could not be represented to and from RLIM_INFINITY. This would avoid the need for the new symbols RLIM_SAVED_MAX and RLIM_SAVED_CUR. But such mapping would arguably be a lie, and the resulting information loss would cause unintuitive program behavior, especially in programs running with appropriate privileges needed to raise hard limits.

A rejected alternative proposal was that if getrlimit() could not correctly return a current limit then it should instead return -1 and set errno to EOVERFLOW. But that would result in unnecessary breakage of programs. (Note that this breakage occurs even when no large files are present.) It would also result in malfunction of programs that assume that they are calling getrlimit() properly and so failure "cannot happen". For example, in the 4.4 BSD-Lite distribution, there are at least 15 unchecked calls to getrlimit(). When the 4.4 BSD csh limit function is used to report the current limits, there is no check of the return code and so the reported results can be entirely incorrect. Also, non-superuser programs typically unlimit themselves with:

     getrlimit(RLIMIT_STACK, &rl);
     rl.rlim_cur = rl.rlim_max;
     setrlimit(RLIMIT_STACK, &rl);
If the getrlimit() fails then garbage is passed to setrlimit() which may result in an unwanted and extremely restricted limit. Several utilities that are part of the GNU C compiler have this problem.

A.2.1.1.14 lio_listio()

The lio_list() function enforces the offset maximum rules since they are logically equivalent to aio_read() and aio_write() which enforce it.

A.2.1.1.15 mmap()

For consistency with read() and write(), the mmap() function will fail when the request extends beyond the offset maximum.

A.2.1.1.16 open()

The open() function called with O_TRUNC set will fail without truncation if the named file is a regular file and the size of the file cannot be represented correctly in an object of type off_t. (See A.2.1.1.4 creat()).

A.2.1.1.17 read(), readv(), write() and writev()

These functions may do a "partial read or write" due to the offset maximum. That is, the value returned may be less than nbyte if the number of bytes remaining which may be transferred is less than nbyte.

A.2.1.1.18 ulimit()

The ulimit() function will return an unspecified result if the result cannot be represented correctly in an object of type long. As this function is already obsolescent, the use of getrlimit() and setrlimit() is recommended for getting and setting process limits.

A.2.2 Changes to CAE Specification Commands and Utilities, Issue 4, Version 2

A.2.2.1 General Porting Suggestions

When porting a program to be large file capable, general areas of concern in addition to the issues mentioned in A.1.1.4 Holes in the Protection Mechanism include:

A.2.2.1.1 Command Line Arguments

Numeric arguments which are file size related, such as a file offset or block count, need to be handled as an appropriately large type. Converting arguments into an off_t that is larger than a long may need to be accomplished with non-standard scanf() formats, if available, or with portable user-written functions that convert ASCII to a large off_t analogous to the strtol() function.

A.2.2.1.2 API Conversion

The program should be recompiled in a large off_t environment or, alternatively, should be converted to use the transitional API. In either case the source must be scanned for the functions listed in 3.1.1.1 64-bit Versions of Interfaces and the data types listed in 3.1.2.1 64-bit Versions of Headers to ensure that all types are properly converted.

A.2.2.1.3 Type Conversion

Whenever a new 64-bit function is used, the argument types and function result will need to be converted as appropriate. Whenever a variable's type is converted (whether via the large off_t compilation environment or the transitional API), all uses of the variable must be checked to determine if further type conversions are warranted. For example, wherever there is a struct stat, all uses of st_size must be checked. If the st_size value is assigned or compared with a variable "v" the variable "v" must be converted if necessary and all uses of "v" must in turn be checked. This is also true of type conversions required for command line arguments.

In addition, the program needs to be checked for file size related variables such as offsets, line numbers, and block counts that must be converted to a large off_t or related type. These variables typically appear inside loops that are performing input and/or output.

A.2.2.1.4 Output Formatting

Output of types that have been converted will probably involve using a different printf() format or using a revised user-written conversion routine. Since there is a larger range of values which take up more space, revision of the output layout may be required.

A.2.2.1.5 Fixed Format Media Issues

Current implementations of the tar and cpio utilities are defective in their support of arbitrarily large files. The pax utility is also equally defective, but is the subject of a proposal in ballot. (See 2.3.3 The pax Utility for discussion of this topic.)

Vendor and third-party backup software is also unable to support large files and will require modification in order to do so.

A.2.2.1.6 Other Languages

This specification is for the C language only. Other languages have different support requirements. For example, the Fortran I/O API has a limit on the number of records, not bytes.

A.2.2.2 Considerations for Utilities in Support of Files of Arbitrary Size

The utilities listed in 2.3.1 Considerations for Utilities in Support of Files of Arbitrary Size are utilities which are used to perform administrative tasks such as to create, move, copy, remove, change the permissions, or measure the resources of a file. They are useful both as end-user tools and as utilities invoked by applications during software installation and operation.

Typical core utilities must be compiled in a "large" off_t compilation environment or must use the transitional APIs. Using the compilation environment reduces the number of editing changes required to port a program, but it does not reduce the effort required to ensure the correctness of the port.

The chgrp, chmod, chown, ln, and rm utilities probably require use of large file capable versions of stat(), lstat(), ftw(), and the stat structure.

The cat, cksum, cmp, cp, dd, mv, sum, and touch utilities probably require use of large file capable versions of creat(), open(), and fopen().

The cat, cksum, cmp, dd, df, du, ls, and sum utilities may require writing large integer values. For example,

The dd, find and test utilities may need to interpret command arguments that contain 64-bit values. For dd the arguments include skip=n, seek=n, and count=n. For find the arguments include -size n. For test the arguments are those associated with algebraic comparisons.

The df utility might need to access large file systems with statvfs().

The ulimit utility will need to use large file capable versions of getrlimit() and setrlimit() and be able to read and write large integer values.

Conversion between off_t (or other derived types) and ASCII is unspecified, which is a significant practical deficiency. This is being considered by other groups. For example, see: ftp://ftp.dmk.com/DMK/sc22wg14/c9x/extended-integers/

A.2.2.3 Additional Requirements for the sh Utility - Porting Recommendations

Pathname expansion (e.g. expanding */foo.c to a/foo.c b/foo.c c/foo.c) and pathname completion might in some cases use the stat() function which would need to be large file capable.

The offset maximum used for shell input and output redirections is implementation-specific. Some vendors prefer to use the smallest supported off_t, others prefer the largest.

A.3 Transitional Extensions to the Single UNIX Specification

A.3.1 Transitional Extensions to CAE Specification System Interfaces and Headers, Issue 4, Version 2

Prior experience with transitional access is reported by SGI, Convex,
(http://www.sas.com/standards/large.file/background) and Programmed Logic Corporation
(http://www.sas.com/standards/large.file/proposals).

A.3.1.1 Transitional Extensions to System Interfaces

A.3.1.1.1 fcntl()

The O_LARGEFILE flag may be set or cleared with F_SETFL. An incorrectly written program may inadvertently clear this flag. For example, some programs put a file into append mode with:
      fcntl(fd, F_SETFL, O_APPEND);
This is incorrect because it turns off all the other open flags, including O_LARGEFILE. Instead, to turn on append mode one should first use F_GETFL to get the current flags:
     int oflag = fcntl(fd, F_GETFL, 0);
then include O_APPEND in the flags:
     oflag |= O_APPEND;
and then set the new flags:
     fcntl(fd, F_SETFL, oflag);
A more complete example would also check for fcntl() failures.

A.3.1.1.2 No fcntl64()

A rejected alternative to extending fcntl() with F_GETLK64 (and so on) would be to specify fcntl64() with F_GETLK (and so on). The former has prior art and less functional redundancy, whereas the latter is more consistent with other transitional functions. This specification does not preclude vendors from supplying an fcntl64().

A.3.1.2 Transitional Extensions to Headers

A.3.1.2.1 <aio.h>

The aio control block has an embedded offset which is of type off_t. A large file enabled aio control block needs a 64-bit offset. For consistency with the other transitional interfaces, a new control block with a 64-bit offset is defined. The offset is of the type off64_t.

Since a new control block is needed, new interfaces are required for all of the existing aio interfaces since every one takes a pointer to the control block as an argument.

A.3.1.2.2 <sys/resource.h>

This proposal does not specify any particular value for RLIM64_INFINITY, RLIM64_SAVED_MAX or RLIM64_SAVED_CUR. Typical implementations should use the value 0x7FFFFFFFFFFFFFFF or 0xFFFFFFFFFFFFFFFF for RLIM_INFINITY, and it is recommended that RLIM64_SAVED_MAX and RLIM64_SAVED_CUR have similar large values. Even though all limit values will be represented in 64-bit types for a few years, specifying them as distinct values now will reduce compatibility problems in the future when the next transition to a still larger type occurs.

A.3.1.2.3 <sys/types.h>

It is not required that ino64_t be a 64-bit type. However, the NFS version 3 protocol allows for 64-bit file serial numbers. For NFS interoperability with systems making use of 64-bit file serial numbers, 64-bit ino_t support is necessary. DCE also may make use of 64-bit file serial numbers.

A.3.2 Accessing the Transitional Extensions to the SUS

A.3.2.1 Compilation Environment - Visibility of Additions to the API

Applications which use the fseeko() and ftello() interfaces should define _LARGEFILE_SOURCE to be 1, then include <unistd.h> and then test that _LFS_LARGEFILE is 1 to determine if the additional functionality is indeed available. This additional functionality may be available even when _LARGEFILE_SOURCE is not defined, but it will not be available to strictly conforming X/Open programs.

This macro does not affect the size of off_t (see 3.3.3 Mixed API and Compile Environments Within a Single Process).

A.3.2.2 Visibility of Transitional API

Applications which wish to use this transitional functionality should define _LARGEFILE64_SOURCE to be 1, then include <unistd.h>, and then test that _LFS64_LARGEFILE, _LFS64_ASYNCHRONOUS_IO and _LFS64_STDIO are set to 1 to determine if the corresponding transitional functionality is indeed available. This transitional functionality may be available even when _LARGEFILE64_SOURCE is not defined, but it will not be available to strictly conforming X/Open programs.

This macro does not affect the size of off_t (see 3.3.3 Utilities: Optional Method for Specifying the Size of an off_t).

If _LARGEFILE64_SOURCE is defined then _LARGEFILE_SOURCE is implied so it need not also be defined (see 3.3.1 Compilation Environment - Visibility of Additions to the API). Similarly, if _LFS64_LARGEFILE is defined then _LFS_LARGEFILE will be defined so it need not also be tested.

A.3.2.3 Mixed API and Compile Environments within a Single Process

Mixing objects from differing compile environments can be dangerous, since some types have different sizes in the differing environments. The types might be used in a way where the size difference causes problems. A system may disallow this mixing. To avoid these problems, don't mix such objects in the same executable, or at least ensure that data shared between files compiled differently does not use any of the types whose meaning may change.

Mixing the standard and transitional APIs is relatively safe, since data types have the same meaning in every file. This mixing permits a smoother and faster migration to a larger off_t environment, because it permits asynchronous upgrades. For example, it permits libraries to be made large file aware without requiring large file awareness in all the programs which use the library or in all the libraries which the library uses. (This is true both for static and for shared libraries.) This is particularly beneficial for situations in which the system vendor, one or more third-party suppliers, and the end user may all be supplying libraries or other objects that are components of a complete program.

A.3.2.4 Utilities: Optional Method for Specifying the Size an off_t

The LFS_CFLAGS variable is used to obtain implementation- specific compiler options, such as flags and preprocessor variable definitions, so that the compiled program will be using a "large" off_t. Similarly the LFS_LDFLAGS variable supplies link editor options, the LFS_LIBS variable supplies link library names, and the LFS_LINTFLAGS variable supplies lint options.

If the size of off_t is controlled by a preprocessor macro variable then it is recommended that the macro be named _FILE_OFFSET_BITS and be supported as follows:

For POSIX compatibility this method must not be affected by the #undef preprocessor or directive. For example:

     #undef lseek
must not alter the size of type off_t in use for a call to lseek().

The functions that might be affected by this option are listed in 3.1.1.1 64-bit Versions of Interfaces.

The types, structures and symbolic constants that might be affected by this option are listed in 3.1.2.1 64-bit Versions of Headers.

It has been argued that there should be a new mode bit (or "magic number") on executable images to indicate whether or not the application is large file aware. This is not precluded by this specification. However, an argument against it is that it requires significant work. Specifically, kernel, compiler, loader, and library changes are needed. It is unclear how the mode bit would support a large file aware application that makes calls to a non-aware shared library.


Revision Information

23Feb96 Version 1.1

The 23Feb96 changes include:
  1. Unix changed to UNIX throughout
  2. Section 1.5 (Changes and Additions) second bullet (Changes to System Interfaces and Headers) added EFBIG
  3. Section 2.2.1 (Changes to System Interfaces) changed "as a future" to "in a future".
  4. Section 2.2.1.1 (aio_read), 2.2.1.1 (aio_write) and 2.2.1.20 (lio_listio) changed nbyte to aiocbp->aio_nbytes; added "is before the end-of-file and" before "is at or beyond" in the EOVERFLOW error.
  5. Section 2.2.1.1 (aio_read), 2.2.1.1 (aio_write) and 2.2.1.20 (lio_listio) changed "greater than or equal to" to "greater than".
  6. Section 2.2.1.4 (fclose, etc.), 2.2.1.7 (fgetc, etc.) and 2.2.1.11 (fprintf, etc.) changed "write beyond" to "write at or beyond".
  7. Section 2.2.1.20 (lio_listio) prefixed lio_opcode with aiocbp->; changed order of phrases in EOVERFLOW and EFBIG (moved "the aiocbp->aio_lio_opcode is LIO_READ" to the front of the sentences); removed "before EOF" in the EOVERFLOW error condition; added "is before the end-of-file and" before "is greater than or equal to the offset maximum".
  8. Section 2.2.2.6 (sys/types.h) and 3.1.2.11 (sys/types.h) changed "must be" to "are defined as" in the sentences starting "The types..".
  9. Section 3.1.1.1 (64-bit Versions of SUS Interfaces) changed title of section to "64-bit Versions of Interfaces". Changed titles in references to match.
  10. Section A.1.1.4.1 (fcntl) moved into A.3.1.1.1 (fcntl).
  11. Section A.1.1.4 (Holes in the Protection Mechanism) body added.
  12. Section A.1.2.1.1 (Offset Maximum and the 2G-1 File Size Limit) boldfaced "B" in "byte line"; changed "a lseek" to "an lseek"; changed "the resulting pointer to the next offset address will overflow" to "the resulting pointer to the next offset address and the file size itself would overflow"; changed title from "Offset Maximum - 2G-1 File Size Limit" to "Offset Maximum and the 2G-1 File Size Limit"; changed "cannot be performed because" to "cannot be performed at that position because" in the last paragraph.
  13. Section A.2.1.1.4 (creat) changed sample code from if (creat(path, ...) < 0) { to if ((fd = creat(path, ...)) < 0) {.
  14. Section A.2.1.1.6 (fgetpos, etc.) changed "this function" to "these functions" in second paragraph; added paragraph beginning "Another potentially serious..." and all that follows to the end of the section.
  15. Section A.3.1.1 (Transitional Extensions...) changed "B.3.1.1.2" to "A.3.1.1.2" in subsection.
  16. Section A.3.1.1.1 (fcntl) Merged sentence "The O_LARGEFILE flag may be set..." with the sentence "The O_LARGEFILE flag can expose..." moved in from A.1.1.4.1 (fcntl).
  17. Section A.3.2.2 (Visibility of Transitional API) changed "Note that if" to "If" in fourth paragraph.
  18. Section A.3.2.4 (Utilities:...) corrected reference to 3.1.2 in the second to the last paragraph to 3.1.2.1 64-bit Version of Headers.
  19. Table of Contents corrected A.3 and A.3.1 heading titles.

24Feb96 Version 1.2

The 24Feb96 changes include:
  1. Added link to Foreword and section.
  2. Section 1.6 (Conformance) removed list, added text for section.
  3. Section 2.2.1.11 (fprintf) changed "needs" to "needed" in the error text.
  4. Section 3.1.2.12 (unistd.h) added LFS_ASYNCHRONOUS_IO version test macro.

01Mar96 Version 1.3

The 01Mar96 changes include:
  1. Changed "Foreword" to "Acknowledgements".
  2. Added body of Acknowledgements.
  3. Section 1.6 (Conformance) 1st paragraph changed "may fail to" to "need not".
  4. Section 3.3.4, Example 2 changed "had" to "has".
  5. Section A.1.2.1.1 (Offset Maximum...) swapped "-" and ">" in top line.
  6. Section A.2.1.1.4 (creat) corrected reference for fstat.
  7. Section 3.3.3 (Utilities:...) corrected reference for Compilation Environment...

05Mar96 Version 1.4

The 05Mar96 changes include:
  1. Changed Version 1.2 in 01Mar96 revision section to Version 1.3
  2. Added additional contributors in the Acknowledgements.

20Mar96 Version 1.5

The 20Mar96 changes include:
  1. Back by popular demand.... Larger fonts in the PostScript Version!
  2. Section 1.2 (Requirements) In the text for "Be fully compliant to the SUS" changed "conversion to the proposed standard" to "conversion to this proposed standard" in the second from the last paragraph.
  3. Section 1.4 (Concepts) Changed "file is larger" to "file size is larger" and changed "only support" to "support only".
  4. Section 1.6 (Conformance) LOTS of changes. In summary: each statement of conformance ("A conforming implementation...") was separated into individual paragraph and in each the phrases "described in" and "listed in" were changed to "specified in"; the version test macro required for each statement of conformance was added along with a reference to the section where the changes to the interfaces and/or headers is described; in the first statement of conformance parenthesis were added around "except...lio_listio()" for clarity. Also deleted the last paragraph (beginning "Implementations which provide...").
  5. Section 2.2.1.7 (fgetc) Removed extra period at end of EOVERFLOW description.
  6. Section 2.2.1.19 (getrlimit) Changed commas before "otherwise" to semicolons in first and second paragraphs; changed "permit" to "might permit" and "do not" to "might not" in the fourth paragraph.
  7. Section 3.0 (Transitional Extensions...) first paragraph: Added sentence beginning "Version test macros..." after the first sentence ("The interfaces...").
  8. Section 3.1.2.8 (sys/resource.h) Added period after description of RLIM64_INFINITY.
  9. Section 3.1.2.12 (unistd.h) In Version Test Macros section added to description of _LFS_ASYNCHRONOUS_IO beginning with "as specified in..."; added "and 3.1.2.2..." to description of _LFS64_ASYNCHRONOUS_IO; added "3.1.1.2 fcntl()..." to description of _LFS64_LARGEFILE; added "and 3.1.2.6..." to description of _LFS64_STDIO. The last paragraph of A.3.2.2 ("If _LFS64_STDIO...") was moved to 3.1.2.12 as a new paragraph in the description of _LFS64_STDIO. In the description of _LFS_LARGEFILE the phrase "the fseeko() and ftello()" was removed and the text beginning with "as specified in..." through the end of the sentence was added.
  10. Section 3.2.1 (Optional Additional...) Changed criteria to criterion (last word of first paragraph).
  11. Section A.1.1.2 (Open Protection...) Removed comma before "and so on" in the third paragraph.
  12. Section A.1.2.1 (Offset Maximum) Added "it" between "that" and "is believed" in last sentence of the fifth paragraph. Also in the fifth paragraph, changed "only applies" to "applies only".
  13. Section A.2.1.1.13 (getrlimit()...) Added text beginning "These values do not..." through "...is really meant." to the end of the first paragraph.
  14. Section A.2.2.1 (General Porting...) In the first paragraph removed the phrase "there are four" and added "include" at then end of the sentence.
  15. Section A.2.2.3 (Type Conversion...) Removed the last sentence of the last paragraph ("Utilities not directly...").
  16. Section A.3.2.2 (Visibility of...) Second paragraph: added missing parenthesis at end of the sentence. Also moved last paragraph ("If _LFS64_STDIO is not defined...") to section 3.1.2.12 (unistd.h) as an additional paragraph in the _LFS64_STDIO description.
  17. Acknowledgements first paragraph: changed "files sizes" to "file sizes" in two places and changed "at least 2**32-1" to "at most 2^31-1". In the list of contributors changed "Hewlett-Packard Inc." to "Hewlett-Packard Co."; changed "Sun Microsystems Corp." to "Sun Microsystems, Inc."; changed "Srimivasam" to "Srinivasan"; removed Art Herzog from Novell list; removed Carl Zeigler from SAS list; added The Santa Cruz Operation, Inc. contributors. Added "(now with Integrated Computer Solutions, Inc.)" after "Mark Hatch".
  18. General: Changed "define[s,d] XXX as 1" to "define[s,d] XXX to be 1".