Unix Programming - Seekable in-memory file?

Author

Seekable in-memory file?

robert.dodier@gmail.com

2006-03-17, 2:57 am

Hello,

Background: there is a Lisp compiler (GCL) which calls gcc as an
intermediate step in compiling a Lisp program. gcc wants its output
file to be seekable. Compiling small programs is pretty common in
the Lisp world, so constructing an ordinary file on the disk and
working
with that (what GCL does now) is a relatively heavy overhead.
Also, it means that GCL cannot run on a read-only filesystem.

So I am looking into the possibility of constructing a seekable
in-memory file. It seems like that should be straightforward,
but I don't know how to do that. If anyone has any advice,
I would be interested to hear it.

All the best,
Robert Dodier

Barry Margolin

2006-03-17, 2:57 am

In article <1142571059.434646.100850@u72g2000cwu.googlegroups.com>,
robert.dodier@gmail.com wrote:

> Hello,
>
> Background: there is a Lisp compiler (GCL) which calls gcc as an
> intermediate step in compiling a Lisp program. gcc wants its output
> file to be seekable. Compiling small programs is pretty common in
> the Lisp world, so constructing an ordinary file on the disk and
> working
> with that (what GCL does now) is a relatively heavy overhead.
> Also, it means that GCL cannot run on a read-only filesystem.
>
> So I am looking into the possibility of constructing a seekable
> in-memory file. It seems like that should be straightforward,
> but I don't know how to do that. If anyone has any advice,
> I would be interested to hear it.

Many versions of Unix have a "tmpfs" file system type, which is often
used for the /tmp directory. This file system makes use of the virtual
memory system rather than a physical disk partition. It will be paged
out to swap space when necessary, but the recently used portions will be
in RAM.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Nils O. Selåsdal

2006-03-17, 2:57 am

Jordan Abel

2006-03-17, 7:50 am

On 2006-03-17, Nils O. Selåsdal <NOS@Utel.no> wrote:
> robert.dodier@gmail.com wrote:
> You'd then need to implement a "filesystem" in user space, which
> is doable on some OSs, with e.g. FUSE or V9FS and possibly others.
>
> Some have a tmpfs/ramfs that could be used too.
>
> gcc's '-pipe' flag helps get rid of some of the temporary files, but
> perhaps it's best to teach the assembler/linker(which I take it is
> what produces the final output) not to need seeking ?

I doubt it's avoidable with the linker, and given the way most build
environments work it wouldn't be worth the effort

Waldek Hebisch

2006-03-17, 8:53 pm

robert.dodier@gmail.com wrote:
> Hello,
>
> Background: there is a Lisp compiler (GCL) which calls gcc as an
> intermediate step in compiling a Lisp program. gcc wants its output
> file to be seekable. Compiling small programs is pretty common in
> the Lisp world, so constructing an ordinary file on the disk and
> working
> with that (what GCL does now) is a relatively heavy overhead.
> Also, it means that GCL cannot run on a read-only filesystem.
>
> So I am looking into the possibility of constructing a seekable
> in-memory file. It seems like that should be straightforward,
> but I don't know how to do that. If anyone has any advice,
> I would be interested to hear it.
>

I am not sure which overhead you want to eliminate. Standard Unix
filesytem buffers files in RAM, so that if file lives short enough
it will be never written to disk. Some Unices have special filesystem
type (tmpfs) so that even long-living files on that filesytem will
stay in RAM (unless system decides that there is shortage of RAM and
swaps the file out). Beware of NFS: basic NFS specification requires file
server to write file to disk and only than report succes of write operation.
So, there should be small overhead (negligable compared to time needed
by gcc to process your file) as long as your file stays on local
filesystem.

Some unices (most??) allow you to have a ramdisk and put a filesytem
there, so you can have a writable filesytem even if your main storage
device is read-only (BTW, if you have no writeable storage, then most
Unix programs will break down).

Finaly, gcc proper works fine without seekable files. I think that
only assembler needs seekable files. So if you provide you own assembler
or hack gas to use in-memory buffer and output its content you
can eliminate files.

--
Waldek Hebisch
hebisch@math.uni.wroc.pl

Barry Margolin

2006-03-19, 12:02 pm

In article <dvfl1a$r9j$1@panorama.wcss.wroc.pl>,
Waldek Hebisch <hebisch@math.uni.wroc.pl> wrote:

> I am not sure which overhead you want to eliminate. Standard Unix
> filesytem buffers files in RAM, so that if file lives short enough
> it will be never written to disk.

But they often require that metadata be written to disk frequently. A
memory-based filesystem avoids these forced writes.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***