|
Home > Archive > Unix Programming > March 2005 > Memory Tables
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
|
|
| Brian C 2005-03-17, 5:55 pm |
| Hello all,
I have a program that uses a shared memory table on AIX (shouldn't
matter the flavor). The table is organized into fixed length records
(lets say 1,024 bytes each). Up until now, I would remove the table at
night and start fresh in the morning, reloading the table from the
mainframe and updating it during the day.
Now, I need to preserve the data as the mainframe cannot reload certain
types of historical data (only as it happens). I figured that there were
two options, each with their own good points and bad points, and I was
wondering if anyone had any input on this.
I was planning at night to dump the table to a file, remove it and
reload the table in the morning (system gets rebooted every Sunday). So,
the options I came up with is this:
1) Simply reload the data via fread().
Good point=Simple.
Bad point=If the record format changes (say I add a field, or even
insert a field), it would have to be re-organized as I would have been
writing to a 1,024 byte record, and the next day the record could be
1,034 bytes.
2) Read in record by record using a map of the old record structure and
new structure, automatically moving data around as needed. The map would
be generated by a custom written application (unless someone knows of
one that exists).
Good point=Only human intervention is making sure the header files
(old and new) are in place.
Bad point=A bit more complicated than option 1. For example, you would
have to find out the size of each individual field, sometimes having to
go into other header files (i.e. pthread.h) to get the size of a field,
etc, etc, etc.
I would really like to do option #2 as I need to write a table editor
for the data, and if I could get something running that could read in
the header files, the table editor would dynamically pick up new fields
as well (less maintenance). However, parsing through system header
files, etc, can cause something to happen down the line (i.e. OS
upgrade, and the C parser I write does not handle a certain comment or
whatever).
Question 1: Is there a way to do this:
char string[]="pthread_mutex_t"; -- and have sizeof() return the
length of what the name in string[] is? Or does it do it at compile time
(this would simplify the process a LOT if it didn't).
Question 2: Any other ideas anyone cares to share?
Thanks in advance.
| |
| those who know me have no need of my name 2005-03-18, 5:56 pm |
| in comp.unix.programmer i read:
> Question 1: Is there a way to do this:
> char string[]="pthread_mutex_t"; -- and have sizeof()
> return the length of what the name in string[] is? Or
> does it do it at compile time
sizeof is an operator, which is resolved at compile-time. (except in the
case of vla's, but that doesn't help you.) recompile the tables as
necessary, comparing the new sizes against the sizes stored along with the
dump.
--
a signature
| |
|
|
Brian C wrote:
> Hello all,
> I have a program that uses a shared memory table on AIX (shouldn't
> matter the flavor). The table is organized into fixed length records
> (lets say 1,024 bytes each). Up until now, I would remove the table
at
> night and start fresh in the morning, reloading the table from the
> mainframe and updating it during the day.
> Now, I need to preserve the data as the mainframe cannot reload
certain
> types of historical data (only as it happens). I figured that there
were
> two options, each with their own good points and bad points, and I
was
> wondering if anyone had any input on this.
> I was planning at night to dump the table to a file, remove it and
> reload the table in the morning (system gets rebooted every Sunday).
So,
> the options I came up with is this:
>
> 1) Simply reload the data via fread().
> Good point=Simple.
> Bad point=If the record format changes (say I add a field, or even
> insert a field), it would have to be re-organized as I would have
been
> writing to a 1,024 byte record, and the next day the record could be
> 1,034 bytes.
>
> 2) Read in record by record using a map of the old record structure
and
> new structure, automatically moving data around as needed. The map
would
> be generated by a custom written application (unless someone knows of
> one that exists).
> Good point=Only human intervention is making sure the header files
> (old and new) are in place.
> Bad point=A bit more complicated than option 1. For example, you
would
> have to find out the size of each individual field, sometimes having
to
> go into other header files (i.e. pthread.h) to get the size of a
field,
> etc, etc, etc.
>
> I would really like to do option #2 as I need to write a table
editor
> for the data, and if I could get something running that could read in
> the header files, the table editor would dynamically pick up new
fields
> as well (less maintenance). However, parsing through system header
> files, etc, can cause something to happen down the line (i.e. OS
> upgrade, and the C parser I write does not handle a certain comment
or
> whatever).
>
> Question 1: Is there a way to do this:
> char string[]="pthread_mutex_t"; -- and have sizeof() return the
> length of what the name in string[] is? Or does it do it at compile
time
> (this would simplify the process a LOT if it didn't).
>
> Question 2: Any other ideas anyone cares to share?
>
> Thanks in advance.
If I had to deal with this, I would probably roll the data out to an
XML file with schema. This would decouple the data from the memory
record format and you may also be able to use some off-the-shelf XML
record editor (or a variant thereof) to modify the data in the XML
file.
The loader could then reload the XML data to the "binary record format
of the day".
Joe
| |
| Michael B Allen 2005-03-19, 2:47 am |
| On Thu, 17 Mar 2005 18:36:57 -0500, Brian C wrote:
> 1) Simply reload the data via fread().
> Good point=Simple.
> Bad point=If the record format changes (say I add a field, or even
> insert a field), it would have to be re-organized as I would have been
> writing to a 1,024 byte record, and the next day the record could be
> 1,034 bytes.
Load and store the data from text converting to or from binary as
necessary. Comma Separated Values (CSV) is good. The tools are simple
[1] and you can reorder fields using a spreadsheet or simple utility
program. Perhaps more important, the next guy that inherits the code
will really appreaciate something easy to understand and adapt.
Mike
[1] http://www.io plex.com/~miallen/libmba/dl/src/csv.c
| |
| Brian C 2005-03-19, 8:51 pm |
| those who know me have no need of my name wrote:
> in comp.unix.programmer i read:
>
>
>
>
> sizeof is an operator, which is resolved at compile-time. (except in the
> case of vla's, but that doesn't help you.) recompile the tables as
> necessary, comparing the new sizes against the sizes stored along with the
> dump.
>
Yes, one option is to simply append all new fields to the end of the
record, figure out that the new record is a different size and read in
accordingly. However, if you delete a field in the middle of the record,
this will not work. Also, I don't like what it would do to the record.
For example, if you have:
typedef struct
{
double Base;
double Net;
double Lal;
...
} CommissionInfo;
typedef struct
{
char Key[512];
int System;
int Inventory;
double NetPnl;
double GrossPnl;
...
CommissionInfo Daily;
CommissionInfo Monthly;
CommissionInfo Yearly;
...
} MyRecord;
If you were to add say a field "TotalNetCommission" that should be
included in "CommissionInfo", you'd have to append three items to the
bottom of the record, thus messing up the groupings.
| |
| Brian C 2005-03-19, 8:51 pm |
| Joe S wrote:
> Brian C wrote:
>
>
> at
>
>
> certain
>
>
> were
>
>
> was
>
>
> So,
>
>
> been
>
>
> and
>
>
> would
>
>
>
>
> would
>
>
> to
>
>
> field,
>
>
> editor
>
>
>
>
> fields
>
>
> or
>
>
> time
>
>
>
>
> If I had to deal with this, I would probably roll the data out to an
> XML file with schema. This would decouple the data from the memory
> record format and you may also be able to use some off-the-shelf XML
> record editor (or a variant thereof) to modify the data in the XML
> file.
>
> The loader could then reload the XML data to the "binary record format
> of the day".
>
>
> Joe
>
Yes, right now, the mainframe sends updates into the program via XML,
and the program parses and records the data. It then goes and generates
different aggregate records based on the update.
This is a good option, however, the only thing that worries me is that
whenever you add a field, you need to add it in three spots (the spot
for the intra-day update, then the reloading part, then the dump part).
If you forget it in the dump part, the data is lost. I mean, you
shouldn't forget, however we're all human after all.
However, you could do an fwrite() of the shared memory segment and then
a seperate process can go and convert it to XML. Then, compress and
backup the fwrite() dump for about a week or so. If you forgot a field,
or say you used a "%s" instead of a "%d", or whatever, hopefully someone
would notice w/in a week. Then, you'd have to replay all the intra-day
updates back into the program where it starts up w/ the last "good"
table dump.
|
|
|
|
|