Unix Programming

Author

pipes

CptDondo

2007-08-28, 7:21 pm

I have a basic question about pipes...

I have an app that will read an instrument about 10x a second. The data
read will take about 12 bytes.

It would be easiest for me from a programming standpoint write the app
so it simply dumps the data into a pipe, and any other app that needs
the data will read it from the pipe.

It's quite possible that this app might be dumping data into the pipe
for days without anyone reading it.

The data is realtime; once it's read it has no time value at all, so I
really don't care if it goes into a bit bucket.

Is it possible to set up a pipe so that the data just "falls out" if no
one is reading it? Or do I have the possibility of a buffer overflow
that would cause a crash somewhere?

Thanks,

--Yan

Maxim Yegorushkin

2007-08-28, 7:21 pm

On Aug 28, 11:26 pm, CptDondo <y...@NsOeSiPnAeMr.com> wrote:

> I have an app that will read an instrument about 10x a second. The data
> read will take about 12 bytes.
>
> It would be easiest for me from a programming standpoint write the app
> so it simply dumps the data into a pipe, and any other app that needs
> the data will read it from the pipe.
>
> It's quite possible that this app might be dumping data into the pipe
> for days without anyone reading it.
>
> The data is realtime; once it's read it has no time value at all, so I
> really don't care if it goes into a bit bucket.
>
> Is it possible to set up a pipe so that the data just "falls out" if no
> one is reading it?

No.

> Or do I have the possibility of a buffer overflow
> that would cause a crash somewhere?

No crash. A pipe is essentially a buffer in kernel space. Once it is
full, write() will block.

For your task you could use UDP multicast datagrams. They just get
discarded if there are no processes interested in receiving it.

CptDondo

2007-08-28, 7:21 pm

Maxim Yegorushkin wrote:
> On Aug 28, 11:26 pm, CptDondo <y...@NsOeSiPnAeMr.com> wrote:
>
[vbcol=seagreen]
>
> For your task you could use UDP multicast datagrams. They just get
> discarded if there are no processes interested in receiving it.
>

Thanks, but UDP won't work.... I have a real possibility of a congested
network, and I am already jumping through hoops to reduce traffic...
(The system can have as many as 600 nodes on a single 802.11b network...
So we're trying to minimize network traffic.)

I'm hoping for something simple, reliable, and easy to maintain.

Alex Colvin

2007-08-29, 1:15 am

>It's quite possible that this app might be dumping data into the pipe
>for days without anyone reading it.

>The data is realtime; once it's read it has no time value at all, so I
>really don't care if it goes into a bit bucket.

This has been called the update/display problem -- you have a process that
updates (writes) the current state, and a process that displays (reads)
the current state. In the simplest case, all you need is the single most
recent state. You want to decouple thew update and display processes. A
pipe (or any sequential communication) requires that they run at the same
speed.

Shared memory is ideal for this kind of communication. The updater writes
the latest status, the displayer reads it. You can optimize a little by
signaling the displayer that there's new data, as long as signals don't
get queued (RTOSs often have "event"s). You may also have to make the
updates atomic, although by clever use of update counters you can just
retry interrupted displaying.

Look up the use of mmap on files, and maybe flock.

--
mac the naïf

Barry Margolin

2007-08-29, 1:15 am

In article <1188341619.047631.241590@w3g2000hsg.googlegroups.com>,
Maxim Yegorushkin <maxim.yegorushkin@gmail.com> wrote:

> On Aug 28, 11:26 pm, CptDondo <y...@NsOeSiPnAeMr.com> wrote:
>
>
> No.
>
>
> No crash. A pipe is essentially a buffer in kernel space. Once it is
> full, write() will block.

It's even worse. If no other process has opened the pipe in read mode,
your open in write mode will block, and you never get to the point of
trying to write. You can specify O_NONBLOCK to prevent the open from
blocking, but you still can't start writing until there's a reader.

A pipe is a connection between two processes -- you can't do anything
with it until there's a process at each end. You can work around this
by having the writing process also open it for reading (some versions of
Unix allow you to use O_RDWR to do it in one step), but that's a bit of
a kludge.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

jellybean stonerfish

2007-08-29, 7:17 am

On Tue, 28 Aug 2007 15:26:06 -0700, CptDondo wrote:

> I have a basic question about pipes...
>
> I have an app that will read an instrument about 10x a second. The data
> read will take about 12 bytes.
>
> It would be easiest for me from a programming standpoint write the app
> so it simply dumps the data into a pipe, and any other app that needs
> the data will read it from the pipe.
>
> It's quite possible that this app might be dumping data into the pipe
> for days without anyone reading it.
>
> The data is realtime; once it's read it has no time value at all, so I
> really don't care if it goes into a bit bucket.
>
> Is it possible to set up a pipe so that the data just "falls out" if no
> one is reading it? Or do I have the possibility of a buffer overflow
> that would cause a crash somewhere?
>
> Thanks,
>
> --Yan

Can't you just write the data to a file? A stupid little loop to
demonstrate.

while true; do date > currenttime ; sleep .1s; done

currenttime is cleared before each write. This makes currenttime only
contain the latest data.
Of course if you access currenttime between the moment that it is cleared
and the moment it is written to it will contain nothing. But that should
be easy to check for in your external program or script. This could make
your hard drive work very much. But,,,,

If you mount a very small ramdisk, and your file is on it, then you will
not be writing data and erasing it constantly on your hard drive.

Does that help in any way?
Of course there would be some details to work out.

stonerfish

Ivan Gotovchits

2007-08-29, 7:17 am

CptDondo wrote:

> Maxim Yegorushkin wrote:
>
>
> Thanks, but UDP won't work.... I have a real possibility of a congested
> network, and I am already jumping through hoops to reduce traffic...
> (The system can have as many as 600 nodes on a single 802.11b network...
> So we're trying to minimize network traffic.)
>
> I'm hoping for something simple, reliable, and easy to maintain.
Then try AF_UNIX datagram socket(SOCK_DGRAM)

Maxim Yegorushkin

2007-08-29, 7:17 am

On Aug 29, 12:49 am, CptDondo <y...@NsOeSiPnAeMr.com> wrote:
> Maxim Yegorushkin wrote:
>
>
>
>
>
>
> Thanks, but UDP won't work.... I have a real possibility of a congested
> network, and I am already jumping through hoops to reduce traffic...
> (The system can have as many as 600 nodes on a single 802.11b network...
> So we're trying to minimize network traffic.)

If both the writing and the reading processes are on the same host
(since you were going to use a pipe), you just use the loopback
interface, which does not involve sending anything over the wire.

Maxim Yegorushkin

2007-08-29, 7:17 am

On Aug 29, 9:48 am, Ivan Gotovchits <ivan.gotovch...@auriga.ru> wrote:
> CptDondo wrote:
>
>
>
>
>
>
>
>
> Then try AF_UNIX datagram socket(SOCK_DGRAM)

This would be very much like a pipe. AF_UNIX sockets must be connected
before sending anything.

Ivan Gotovchits

2007-08-29, 7:17 am

Maxim Yegorushkin wrote:

> On Aug 29, 9:48 am, Ivan Gotovchits <ivan.gotovch...@auriga.ru> wrote:
>
> This would be very much like a pipe. AF_UNIX sockets must be connected
> before sending anything.
I think that only stream sockets must be connected, not datagram, but I'm
not sure. I'll too prefer using "normal" PF_INET sockets, it is simplier
and more portable.

Rainer Weikusat

2007-08-29, 7:17 am

CptDondo <yan@NsOeSiPnAeMr.com> writes:
> I have a basic question about pipes...
> I have an app that will read an instrument about 10x a second. The
> data read will take about 12 bytes.
>
> It would be easiest for me from a programming standpoint write the app
> so it simply dumps the data into a pipe, and any other app that needs
> the data will read it from the pipe.
>
> It's quite possible that this app might be dumping data into the pipe
> for days without anyone reading it.
>
> The data is realtime; once it's read it has no time value at all, so I
> really don't care if it goes into a bit bucket.
>
> Is it possible to set up a pipe so that the data just "falls out" if
> no one is reading it?

You could use an unconnected PF_UNIX datagram socket and sendto to
send the updates to some AF_UNIX destination address. If someone cares
(ie another process has a PF_UNIX datagram socket bound to this
address), that someone will be able to receive the data, otherwise,
the sending process will get some error (eg ECONNREFUSED or ENOENT)
which it can ignore.

But not using reliable communication may be a better solution, cf Alex
Colvin's post.

Rainer Weikusat

2007-08-29, 7:17 am

Maxim Yegorushkin <maxim.yegorushkin@gmail.com> writes:
> On Aug 29, 9:48 am, Ivan Gotovchits <ivan.gotovch...@auriga.ru> wrote:

[...]

>
> This would be very much like a pipe. AF_UNIX sockets must be connected
> before sending anything.

Datagram sockets are always connection-less.

Maxim Yegorushkin

2007-08-29, 7:17 am

On Aug 29, 11:18 am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
> Maxim Yegorushkin <maxim.yegorush...@gmail.com> writes:
>
> [...]
>
>
>
> Datagram sockets are always connection-less.

Sorry, my mistake, forgot about sendto().

CptDondo

2007-08-29, 1:20 pm

Alex Colvin wrote:
>
>
>
> This has been called the update/display problem -- you have a process that
> updates (writes) the current state, and a process that displays (reads)
> the current state. In the simplest case, all you need is the single most
> recent state. You want to decouple thew update and display processes. A
> pipe (or any sequential communication) requires that they run at the same
> speed.
>
> Shared memory is ideal for this kind of communication. The updater writes
> the latest status, the displayer reads it. You can optimize a little by
> signaling the displayer that there's new data, as long as signals don't
> get queued (RTOSs often have "event"s). You may also have to make the
> updates atomic, although by clever use of update counters you can just
> retry interrupted displaying.
>
> Look up the use of mmap on files, and maybe flock.
>

Thanks for the great discussion, folks.

I thought about shared memory, but I've never used it before.

Reading the various shared memory HOWTOs got me a bit confused.

From what I gather, I'm better off using BSD mmap, rather than SysV
IPC... Right?

And in essence, all I have to do on the server is something like:

fd = open(somefile,O_WRONLY);
mmap(some block, PROT_WRITE....,fd)

then write to fd and do a msync????

On the client,

fd = open(somefile,O_RDONLY);
mmap(some block, PROT_READ....,fd)

then read from fd...

Is that about it?

I can't find any examples for this sort of thing. I guess it's too
simple. :-)

Thanks,

--Yan

Rainer Weikusat

2007-08-29, 7:26 pm

CptDondo <yan@NsOeSiPnAeMr.com> writes:

[...]

> From what I gather, I'm better off using BSD mmap, rather than SysV
> IPC... Right?

I think mmap is easier to use.

> And in essence, all I have to do on the server is something like:
>
> fd = open(somefile,O_WRONLY);
> mmap(some block, PROT_WRITE....,fd)
>
> then write to fd and do a msync????

Assuming a suitable file exists:

fd = open(somefile, O_RDWR);
p = mmap(NULL, length_of_data, PROT_READ | PROT_WRITE, MAP_SHARED,
fd, 0);
close(fd)

Afterwards, you can just use the memory area pointed to by 'p' like
any other memory area.

> On the client,
>
> fd = open(somefile,O_RDONLY);
> mmap(some block, PROT_READ....,fd)

fd = open(somefile, O_RDONLY);
p = mmap(NULL, length_of_data, PROT_READ, MAP_SHARED, fd, 0);
close(fd);

Both 'client p' and 'server p' will point the same area of physical
memory from now on. 'msync' is only needed if you want to ensure that
changes are written to disk. You will need to come up with some sort
of 'access protocol' to ensure that the client doesn't read from the
same part of this area the server is currently writing to.

CptDondo

2007-08-29, 7:26 pm

Rainer Weikusat wrote:
> CptDondo <yan@NsOeSiPnAeMr.com> writes:
>
> [...]
>
>
> I think mmap is easier to use.
>
>
> Assuming a suitable file exists:
>
> fd = open(somefile, O_RDWR);
> p = mmap(NULL, length_of_data, PROT_READ | PROT_WRITE, MAP_SHARED,
> fd, 0);
> close(fd)
>
> Afterwards, you can just use the memory area pointed to by 'p' like
> any other memory area.
>
>
> fd = open(somefile, O_RDONLY);
> p = mmap(NULL, length_of_data, PROT_READ, MAP_SHARED, fd, 0);
> close(fd);
>
> Both 'client p' and 'server p' will point the same area of physical
> memory from now on. 'msync' is only needed if you want to ensure that
> changes are written to disk. You will need to come up with some sort
> of 'access protocol' to ensure that the client doesn't read from the
> same part of this area the server is currently writing to.
>

That's too easy... :-)

One last question about the "real time" aspect of this:

I only have one process that would write to this area. So if in the
server I set

p[0] = -1;
write ...
p[0] = 0;

then the client would know not to try to read the area until p[0] == 0.
Would this work? Or are the writes buffered somehow?

Thanks!

--Yan

moi

2007-08-29, 7:26 pm

On Wed, 29 Aug 2007 12:39:48 -0700, CptDondo wrote:

> Rainer Weikusat wrote:

>
> That's too easy... :-)
>
> One last question about the "real time" aspect of this:
>
> I only have one process that would write to this area. So if in the
> server I set
>
> p[0] = -1;
> write ...

This "write" does not exist; at least the programs can't
detect whether is happened or not. The programs just "peek" into the same
diskbuffer. Whether that diskbuffer has been written to the disk or not is
irrelevant.

> p[0] = 0;
>
> then the client would know not to try to read the area until p[0] == 0.
> Would this work? Or are the writes buffered somehow?
>

No. The only possible buffering is done by the compiler. This is what the
'volatile' keyword is all about. AFAIK, writes of machine-words are atomic.
Reads/writes to disks are atomic, too; (modulo pagesize). This can only be
violated by a hardware crash.

So, you are safe if there is a flag or pointer that is writable
by only the writer-process. (See also Lamport
http://research.microsoft.com/users...ubs.html#bakery
or http://research.microsoft.com/users...pubs.html#rd-wr )

HTH,
AvK

CptDondo

2007-08-29, 7:26 pm

moi wrote:
> On Wed, 29 Aug 2007 12:39:48 -0700, CptDondo wrote:
>
>
>
> This "write" does not exist; at least the programs can't
> detect whether is happened or not. The programs just "peek" into the same
> diskbuffer. Whether that diskbuffer has been written to the disk or not is
> irrelevant.
>
>
>
> No. The only possible buffering is done by the compiler. This is what the
> 'volatile' keyword is all about. AFAIK, writes of machine-words are atomic.
> Reads/writes to disks are atomic, too; (modulo pagesize). This can only be
> violated by a hardware crash.
>
> So, you are safe if there is a flag or pointer that is writable
> by only the writer-process. (See also Lamport
> http://research.microsoft.com/users...ubs.html#bakery
> or http://research.microsoft.com/users...pubs.html#rd-wr )
>

Great, thanks!

--Yan

fjblurt@yahoo.com

2007-08-29, 7:26 pm

On Aug 29, 12:39 pm, CptDondo <y...@NsOeSiPnAeMr.com> wrote:
> Rainer Weikusat wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> That's too easy... :-)
>
> One last question about the "real time" aspect of this:
>
> I only have one process that would write to this area. So if in the
> server I set
>
> p[0] = -1;
> write ...
> p[0] = 0;
>
> then the client would know not to try to read the area until p[0] == 0.
> Would this work? Or are the writes buffered somehow?

It won't work, but not for the reason you think. The server may start
writing in between the client checking the flag and finishing the
reading. That is:

client sees p[0]==0
client reads first three bytes
CONTEXT SWITCH
server sets p[0]=-1
server writes all bytes
server sets p[0]=0
CONTEXT SWITCH
client reads remaining bytes, which don't match up with the first
three.

This is the sort of race condition that you must beware when dealing
with interprocess communication. It's especially insidious because
most of the time it won't occur, so your program will fail at random
and be very difficult to debug.

You can try and think of clever ways to avoid the scenario I mention,
but it's difficult to ensure that you've got all of them. For
instance, on some systems it might be possible for a context switch to
occur in the *middle* of a statement like "p[0]=0", with peculiar
results.

A robust approach is to lock the file when reading or writing the
data. See section 2.5 of the FAQ for this newsgroup for information
on file locking. Since file locking is designed for exactly this sort
of situation, you can be sure that it will be free of races, as long
as you use it on every access.

Logan Shaw

2007-08-30, 1:19 am

fjblurt@yahoo.com wrote:
> On Aug 29, 12:39 pm, CptDondo <y...@NsOeSiPnAeMr.com> wrote:

>
> It won't work, but not for the reason you think. The server may start
> writing in between the client checking the flag and finishing the
> reading.

> A robust approach is to lock the file when reading or writing the
> data. See section 2.5 of the FAQ for this newsgroup for information
> on file locking. Since file locking is designed for exactly this sort
> of situation, you can be sure that it will be free of races, as long
> as you use it on every access.

Correct me if I'm wrong, but if you need more fine-grained locking, you
should be able to use sem_init() to create any number of semaphores in
the shared memory area.

That way the data collection routine could write a history of several
samples into the area, and a client could read them at its leisure.
And you could have a separate lock for each one, thus reducing the
chances that the client and server would need the lock at the same time.
The obvious way to do this would be with a circular buffer.

(This would be helpful if the client need to collect the data and
write it to a log file or something; it wouldn't have to worry much
about running super frequently so that it doesn't miss data.)

- Logan

Logan Shaw

2007-08-30, 1:19 am

jellybean stonerfish wrote:
> Can't you just write the data to a file? A stupid little loop to
> demonstrate.
>
> while true; do date > currenttime ; sleep .1s; done
>
> currenttime is cleared before each write. This makes currenttime only
> contain the latest data.
> Of course if you access currenttime between the moment that it is cleared
> and the moment it is written to it will contain nothing. But that should
> be easy to check for in your external program or script. This could make
> your hard drive work very much.

If you're looking for a quick and dirty solution like this, and if you
don't mind being a bit wasteful, you can easily eliminate the race condition:

while :
do
collect_data > new_file
mv new_file current_file
sleep 1
done

If the client then reads current_file, it will always see a consistent
view. This is because new_file will always be a new file with a new
i-node number, and renaming new_file to current_file will not overwrite
current_file but will instead atomically change the directory entry so
that any processes which still are reading will continue to see their
own file (the one they opened).

- Logan

moi

2007-08-30, 7:20 am

On Wed, 29 Aug 2007 16:50:28 -0700, fjblurt wrote:

> On Aug 29, 12:39 pm, CptDondo <y...@NsOeSiPnAeMr.com> wrote:

>
> It won't work, but not for the reason you think. The server may start
> writing in between the client checking the flag and finishing the
> reading. That is:
>
> client sees p[0]==0
> client reads first three bytes
> CONTEXT SWITCH
> server sets p[0]=-1
> server writes all bytes
> server sets p[0]=0
> CONTEXT SWITCH
> client reads remaining bytes, which don't match up with the first
> three.
>

That is what I meant by "writes of machine words are atomic"
IMO the context switches can't happen in the middle of a bus-tranfer.

(for curing it, check the second Lamport paper)

HTH,
AvK

Golden California Girls

2007-08-30, 7:20 am

moi wrote:
> AFAIK, writes of machine-words are atomic.

That depends on the silicon it is running on. While I don't know of any today
that aren't, don't depend on this for tomorrow.

> Reads/writes to disks are atomic, too; (modulo pagesize). This can only be
> violated by a hardware crash.

This isn't true. There exists hardware where each byte is read by spin loops.
Loops can get interrupted. Also there is hardware where each byte is read by
interrupt. Interrupts can be blocked or higher priority ones arrive. And it
may not be true even for DMA hardware as usually there is a reset command that
can be issued to the hardware that will abort a transfer in progress. Also I
believe you meant to say modulo device sector size. And you have totally
forgotten write with read back check where the read back fails, the driver may
do other things before it tries a rewrite and re check, or it may even spare a
sector.

Captain Dondo

2007-08-30, 1:20 pm

V Thu, 30 Aug 2007 05:10:11 -0700, Golden California Girls napsal(a):

> moi wrote:
>
> That depends on the silicon it is running on. While I don't know of any today
> that aren't, don't depend on this for tomorrow.
>
>
> This isn't true. There exists hardware where each byte is read by spin loops.
> Loops can get interrupted. Also there is hardware where each byte is read by
> interrupt. Interrupts can be blocked or higher priority ones arrive. And it
> may not be true even for DMA hardware as usually there is a reset command that
> can be issued to the hardware that will abort a transfer in progress. Also I
> believe you meant to say modulo device sector size. And you have totally
> forgotten write with read back check where the read back fails, the driver may
> do other things before it tries a rewrite and re check, or it may even spare a
> sector.

OK, please explain this....

All I want/need for this particular case is to:

read a series of sensors
write the values to some place where other apps can read them
do so in a way that guarantees partial reads won't take place

speed isn't an issue here. I'm measuring outdoor temperature for freezing,
so the changes will take place over a period of hours.

I really don't care if I miss a reading.

I am working on an embedded platform; the keypad and the sensors are muxed
so I am reading the sensors between keystrokes at a rate of 1 full sensor
scan every 100 msec.

Will my concept work?

ie.

server:

set up shared memory
mark a byte 'dirty'
write values of sensors into other bytes
mark a byte 'clean'

client:

if a byte is dirty, skip
else read values of sensors

Thanks,

--Yan

Alex Fraser

2007-08-30, 1:20 pm

"moi" <root@localhost.localdomain> wrote in message
news:pan.2007.08.29.22.01.00.176184@localhost.localdomain...
> On Wed, 29 Aug 2007 12:39:48 -0700, CptDondo wrote:
[snip]
>
> This "write" does not exist; at least the programs can't
> detect whether is happened or not. The programs just "peek" into the same
> diskbuffer. Whether that diskbuffer has been written to the disk or not
> is irrelevant.
>
>
> No. The only possible buffering is done by the compiler. This is what the
> 'volatile' keyword is all about.

There is also often buffering and reordering in the CPU which makes a scheme
like the above potentially unreliable on multi-processor systems. Google
"memory barriers".

Alex

Bahadir

2007-08-30, 1:20 pm

On 28 Aug, 23:26, CptDondo <y...@NsOeSiPnAeMr.com> wrote:
> I have a basic question about pipes...
>
> I have an app that will read an instrument about 10x a second. The data
> read will take about 12 bytes.
>
> It would be easiest for me from a programming standpoint write the app
> so it simply dumps the data into a pipe, and any other app that needs
> the data will read it from the pipe.
>
> It's quite possible that this app might be dumping data into the pipe
> for days without anyone reading it.
>
> The data is realtime; once it's read it has no time value at all, so I
> really don't care if it goes into a bit bucket.
>
> Is it possible to set up a pipe so that the data just "falls out" if no
> one is reading it? Or do I have the possibility of a buffer overflow
> that would cause a crash somewhere?
>
> Thanks,
>
> --Yan

I think you must ask yourself the question *when* the data should be
discarded, if noone reads it. Then the writer could truncate the file,
pipe or any other data channel at these specified intervals. If you
use a pipe, you could also use a file, which would greatly increase
your write interval if you have lots of storage.

Thanks,
Bahadir

fjblurt@yahoo.com

2007-08-30, 7:21 pm

On Aug 29, 6:56 pm, Logan Shaw <lshaw-use...@austin.rr.com> wrote:
> jellybean stonerfish wrote:
>
>
>
> If you're looking for a quick and dirty solution like this, and if you
> don't mind being a bit wasteful, you can easily eliminate the race condition:
>
> while :
> do
> collect_data > new_file
> mv new_file current_file
> sleep 1
> done
>
> If the client then reads current_file, it will always see a consistent
> view. This is because new_file will always be a new file with a new
> i-node number, and renaming new_file to current_file will not overwrite
> current_file but will instead atomically change the directory entry so
> that any processes which still are reading will continue to see their
> own file (the one they opened).
>
> - Logan

That's a good idea. Given that the data is only a few bytes and only
needs to be updated a few times a second, this should work fine and is
much simpler than anything else I've seen proposed.

Alex Colvin

2007-08-31, 1:18 am

If you're updating more than an atomic write, and if flock() doesn't do it
for you, AND there is a single process doing the writes, then you can code
it as a transaction.

Instead of a single flag , use two counters: ENTERS and EXITS.

The updater (and there must be only one) keeps a sequence number. When
starting an update, increment the sequence number and write it into
ENTERS. Then write the new data. When done, write it into EXITS.
Incrementing the sequence ensures that ENTERS=EXITS except during an
update.

The reader grabs the EXITS count into a temporary (READ) and copies the
relevant data. When done, it compares READ to ENTERS.

READ<ENTERS: another update started during the read
READ>ENTERS: the read started during an update
READ=ENTERS: the read was not interrupted, nor did it interrupt

If READ=ENTERS, you're good. If not, try again.

Note that the updates never have to wait (as they might with flock), but
in exchange the reader may have to retry. With a low duty-cycle, this
should be ok.

Note too, that the reader never touches anything, so you can have many of
them. And they can come and go, (crash and restart) as you please.

Note also, that if the sequence number wraps during the time it takes to
read the data, you hav e real problems. Otherwise, we don't really care
that the sequence number wraps, just that it doesn't repeat.

Note especially that this assumes nothing is re-ordering writes. On a
single processor, this should work.
--
mac the naïf

Alex Colvin

2007-08-31, 1:18 am

>> Look up the use of mmap on files, and maybe flock.

>Thanks for the great discussion, folks.
>I thought about shared memory, but I've never used it before.
>Reading the various shared memory HOWTOs got me a bit confused.

for an update rate of 10Hz, forget about mmap. just rewind and re-read the
file. It will be in the buffer cache anyway.

--
mac the naïf

Rainer Weikusat

2007-08-31, 7:17 am

Alex Colvin <alexc@TheWorld.com> writes:

[...]

> Note also, that if the sequence number wraps during the time it takes to
> read the data, you hav e real problems.

It is easy to deal with that: Calculate two differences, a = ENTERS -
READS and b = READS - ENTERS, using unsigned arithmetic. Then use the
test you gave in your posting with a for READS and b for ENTERS. This
will give correct results if the maximum possible distance from READS
to ENTERS is less than half the maximum value of the (unsigned) type
used to represent both.

Side note: A visualization of this would be a 'clock' where all
integers which belong to the set of possible values for a given
unsigned type are 'in hour positions', starting with zero at
'twelve'. a is the number of clockwise steps necessary to reach ENTERS
from READS and b is the number of counter-clockwise steps needed to
accomplish the same.

Don't laugh if you think this is trivial. At least the person who
wrote the 'original' Kerberos 5 GSSAPI sequence number comparison code
did not understand it and I had to replace that with an algorithm based
on the idea above (and using a 'sequence number window' like the one
described in some IPsec RFC) because the Kerberos 1.4 code I am using
on the (still existing :-> ) appliances of my employer used to throw
wrong 'duplicate token' errors at me.

Alex Colvin

2007-08-31, 7:17 pm

[vbcol=seagreen]
>It is easy to deal with that: Calculate two differences, a = ENTERS -
>READS and b = READS - ENTERS, using unsigned arithmetic. Then use the
>test you gave in your posting with a for READS and b for ENTERS. This
>will give correct results if the maximum possible distance from READS
>to ENTERS is less than half the maximum value of the (unsigned) type
>used to represent both.

Although in this case, we don't care about the (wrapped) sign of the
difference, just whether the difference is nonzero.

Anyway, what I meant is that if reading the data can be preempted long
enough for the sequence to wrap, you probably don't really have a display
of live data. At 10Hz and a 32-bit counter, it would take 12 years.

If your read process can be transparently suspended for 12 years and then
resumed, your system is a lot more stable than mine.

--
mac the naïf