Hi Dudes,
II2.5/0011 patch9563 on axp.osf/00
I've tried to unload a STAR Distributed database with:
unloaddb -c -ulust test_ldb1_ddb/star and received errors.
E_RD0043 No more space in cache to build or extend a descriptor block.
E_UL0104 No more space in the hash table.
E_RD0043 No more space in cache to build or extend a descriptor block.
E_UL0104 No more space in the hash table.
E_PS0F01 The descriptor cache is currently full. Try again later.
A subsequent connection to the database with sql also then received the same
 set
of errors. In both cases thay appeared in the errlog as well. I tried to cle
ar the
problem with trace point rd001 but this proved ineffective.
My RDF Cache is some 22M. I've run some RD Cache usage summary stats
with trace point rd11. These show.
!............................................... !
RDF Cache Memory Available: 19776640 !
RDF memory cache size : 22282240 !
Max number of objects allowed on Cache: !
RELcache: 600, QTREE Cache: 600, !
LDBDesc Cache 0, DEFAULTS cache: 256 !
Hashids: !
RELcache: 1188879872, QTREE Cache: 1188871576, !
LDBDesc Cache 0, DEFAULTS cache: 1188777800
!...............................................
Which seems to indicate the vast majority of the cache memory is unused.
Furthermore, this database (and the several others I was playing with) are p
retty
similar in structure. Yet the sequence of events shows many Distributed
databases unloaded successfully and then these problems began and from then
on no other distributed database would unload. During this period user
connections to non affected databases continued - apparently successfully.
After a bit of thrashing around I became suspicious  about the quality of th
e
RDF Cache memory and so I ran a trace point rd22. Which possibly wasn't the
best idea I've ever had. It immediatly put the following in the errlog.log:
E_RD010D_BAD_CKSUM consistency check - checksum error on infoblk
indicates potential memory corruption
E_RD010D_BA D_CKSUM consistency check - checksum error on infoblk
indicates potential memory corruption
E_PS0904_BA D_RDF_GETDESC RDF error occurred when getting
description for an object.
E_PS0007_IN T_OTHER_FAC_ERR PSF detected an internal error when
calling other facility.
E_SC0215_PS F_ERROR Error returned by PSF.
The rmcmd then failed. At which point I shut down the system. and it restart
ed
okay. Before, I rerun the attempt to unload the star Distributed database, a
re
there any suggestions?
Martin Bowes
--
Random Duckman Quote #78:
Uranus: We spent the entire morning burning your sheets, and the mattress,
and the bed and the carpet around the bed leading into the bathroom,
and the bathroom.

[ Post a follow-up to this message ]

Hi Karl,

From what I've managed to scrape togethor so far there are
three candidates, plus we have to allow for this being in either the
DBMS server or the STAR server. In this case, I think the latter.

DBMS            STAR
rdf_memory       22282240        44564480
rdf_max_tbls       600             1200
rdf_tbl_cols       256              512

Marty
--
Random Farscape Quote #33:
Rygel - Doesn't it strike anybody as a viable policy to simply shoot
them.

[ Post a follow-up to this message ]

Hi Jean Luc,

Ingres normally has around 500 open files on this platform, and
my limits are set to 4096. So I've got a lot of room to move in. I didn't
monitor open files at thre time of the unload, but there is no mention of
a problem in the system messages file.

I'm leaning towards the rd3 or rd10 trace points as a work
around for the moment.

Marty

> Hi Marty,
>
> It's not a dmf cache problem, even if this parameter name is
> confusing. That might be a bad error handling in PSF that causes these
> errors in fact. And you're right, the server must be stopped and
> restarted once the RD0043 is reported. Use preferably trace point
> RD0010 instead of rd001 to flush all RDF caches before hitting the RDF
> full messages (before the unloads).
>
> RD01 Invalidate relation cache
> RD02 Invalidate relation & QTREE cache
> RD03 Invalidate LDB cache (STAR)
> RD04 All of the above
> RD05 Invalidate defaults cache
> RD10 Invalidate everything in RDF
> RD11 RDF cache usage summary, to II_DBMS_LOG
> RD12 RDF cache stats, same as shutdown summary, to II_DBMS_LOG
> RD13 Run a memory consistency check and dump to II_DBMS_LOG
> RD20 Don't look for synonyms
> RD21 (STAR) Don't update iidd_ddb_tableinfo timestamp
> RD22 Compute and use checksums to detect RDF memory corruption
> RD23 (STAR) Print RDF queries to LDB
>
> I don't want to learn you how to suck eggs ;O),
> you are certainly aware about it but:
>
> If it's number of open files related:
> 1/ dmf_hash_size value may be involved.
>
> As I previously wrote, dmf_hash_size is used for hash lookups of Table
> Control Blocks (TCBs) and is also used to control the number of file
> descriptors that can be opened at one time. But keep in mind that
> increasing to much dmf_hash_size can affect performance significantly
> ;O( and table and indexes split over several locations, means more
> open files.
>
> 2/And so the system limits:
> I would check ulimit -ha and ulimit -Ha for ingres
>
> And the command to check # of open files per process is: sysconfig -q
> proc (-Q option gives min and max values for these parameters): ...
> open_max_soft = 4096 open_max_hard = 4096 ... Then if needed, lsof can
> report how many files are opened when the problem occurs.
>
> HTH to run these unloads without error,
> Cheers,
> Uncle Jean-Luc
--
Random Farscape Quote #27:
Rygel - I don't believe the Universe will follow my preconceptions, but I
know a fact when it hits me in the face.

[ Post a follow-up to this message ]

Hi Karl,

Spot on with the cache parameter names! Check the Big Brain
on Brad. Ezekial 25:17 and all that jazz...

rdf_avg_ldbs 5
rdf_cache_ddbs 10

The installation has seen these error messages before (about a
year ago), but in that instance, the parameters I mentioned below were
increased (at least doubled, then doubled again) and a procedure used
to register tables in the distributed databases was altered to use
register ... with refresh.

I suspect that in this case of hammering the system with
unloads of star DDBs and CDBs that we managed to swamp the rdf
cache in a particurly non-standard fashion. This looks reasonable as
I've managed to individually unload some of the databases indicated as
being in trouble in the original group unload.

So I could double each of rdf_avg_ldbs and rdf_cache_ddbs
and then put in a sleep 2 between each unload to give the system
some breathing space. Uncle Jean Luc suggested using trace point
rd10 to clear the cache between unloads.

My tendency is to avoid trace points in code wherever possible.
But this program is not meant to execute regurly - its just a one off to
snapshot the databases before we turn off the box, jack it up onto
wheels and push it a few kilometers up the road.

Marty

On 8 Mar 2005 at 12:42, Karl & Betty Schendel wrote:
 
>
> No, it's some weird star thing.  Try raising rdf_avg_ldbs or
> rdf_cache_ddbs. I think those are the names.
>
> Karl
>
--
Random Farscape Quote #18:
Aeryn - Its my turn, this is my plan!
John  - Thats your plan! Wile. E. Coyote comes up with better plans
than that.

[ Post a follow-up to this message ]