|
Home > Archive > Debian Developers > July 2005 > Shared library versioning
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Shared library versioning
|
|
| Alexis Papadopoulos 2005-07-07, 7:56 am |
| Hello,
I'm actually making some .deb packages out of a single source archive.
One of them should contain a shared library. The upstream author's
package's version is 5.13 and the soname of his library has been set to
513. After having contacted him, he told me that was done because
apparently each time the new version of the library became uncompatible
with the previous ones, the major version should be incremented,
something that was disturbing because the library is often subject to
modifications leading to incompatibilities.
After having searched around on the internet I didn't find any
information on this, but I read once again the SONAME chapter of the
debian library packaging manual and didn't see this "method" of
versioning mentioned.
What I understood so far is that when upgrading a shared library the old
version is left around and only the link libname.so is updated. What
happens now if I update this shared library without recompiling the
software that was linked agains the old version ?
Thanks in advance for your comments
Alexis Papadopoulos
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Paul TBBle Hampson 2005-07-07, 7:56 am |
| On Thu, Jul 07, 2005 at 10:13:20AM +0200, Alexis Papadopoulos wrote:
> Hello,
> I'm actually making some .deb packages out of a single source archive. One of
> them should contain a shared library. The upstream author's package's version
> is 5.13 and the soname of his library has been set to 513. After having
> contacted him, he told me that was done because apparently each time the new
> version of the library became uncompatible with the previous ones, the major
> version should be incremented, something that was disturbing because the
> library is often subject to modifications leading to incompatibilities.
> After having searched around on the internet I didn't find any information on
> this, but I read once again the SONAME chapter of the debian library
> packaging manual and didn't see this "method" of versioning mentioned.
> What I understood so far is that when upgrading a shared library the old
> version is left around and only the link libname.so is updated. What happens
> now if I update this shared library without recompiling the software thatwas
> linked agains the old version ?
The general rules of thumb as I understand them are:
* If the ABI changes in a way that's not backwards-compatible, ie symbols
change meaning, order or are deleted, you have to increment the sover
* If the ABI changes in a way that's backwards-compatible, ie symbols are added
to the end of the list only, you should not increment the sover. You _do_
need to fix the shlibs file in the Debian package, though, since programs
built against this library are not backwards-compatible.
Failing the first means programs stop working mysteriously.
Failing the second means programs have to be recompiled excessively, as they
will refuse to work with an otherwise usable library becaues the sover changed.
Unless the library is in an incredible state of flux, the sover probably ought
to be the major version you mentioned above.
If it _is_ in such an incredible state of flux, and it's not huge, considera
static-only version, ala ffmpeg or xlibs-static. (Although I believe policyhas
some strong words against this path... it introduces major pain for archs that
aren't i386 and PPC revolving around PIC/nonPIC code.)
If the library's in flux, not staticable, and used only by the related
programs, then maybe bundling it all together in one .deb, and putting the
library somewhere below /usr/lib private to that deb (or that source) might
work. This is how FreeRADIUS stores all its private libraries and modules.
I believe there is work underway to make dh_shlibs a little bit cleverer about
working out when you've gotten the rules wrong... Although I can't see how
that'd work without having the previous version of the library at hand. Butfor
the moment, care and caution is the motto. ^_^
--
-----------------------------------------------------------
Paul "TBBle" Hampson, MCSE
8th year CompSci/Asian Studies student, ANU
The Boss, Bubblesworth Pty Ltd (ABN: 51 095 284 361)
Paul.Hampson@Anu.edu.au
"No survivors? Then where do the stories come from I wonder?"
-- Capt. Jack Sparrow, "Pirates of the Caribbean"
This email is licensed to the recipient for non-commercial
use, duplication and distribution.
-----------------------------------------------------------
| |
| Alexis Papadopoulos 2005-07-07, 7:56 am |
| Thanks for that one,
the thing is that the upstream author is using libtool which has a
somehow "special" versioning method. Apparently when the library's
interface changes in a way backwards-compatibility is broken, the major
(what they call current) version must be incremented.
I will have to discuss this with him because it means that some changes
have to made. But isn't this versioning issue a headache ? I mean, if
your sources compile a library and some binaries linked against it, you
have to maintain 2 version informations : the sources' version and the
library's one. Up until now the upstream author simply used a VERSION
file that was incremented "automagically" before CVS commit. Now he must
to keep a second version info for the library that he must supervise
manually in case of backwards-incompatibility...
Anyway, I will have to see with him in order to find a solution because
I think that having a (in libtool's vocabulary) current version of 513
isn't really acceptable...
Alexis Papadopoulos
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Paul TBBle Hampson 2005-07-07, 7:56 am |
| On Thu, Jul 07, 2005 at 10:54:15AM +0200, Alexis Papadopoulos wrote:
> Thanks for that one,
> the thing is that the upstream author is using libtool which has a somehow
> "special" versioning method. Apparently when the library's interface changes
> in a way backwards-compatibility is broken, the major (what they call
> current) version must be incremented.
> I will have to discuss this with him because it means that some changes have
> to made. But isn't this versioning issue a headache ? I mean, if your sources
> compile a library and some binaries linked against it, you have to maintain 2
> version informations : the sources' version and the library's one. Up until
> now the upstream author simply used a VERSION file that was incremented
> "automagically" before CVS commit. Now he must to keep a second version info
> for the library that he must supervise manually in case of
> backwards-incompatibility...
It's a single headache for the one library developer/packager, as
opposed to headaches for _every user_ of the library.
> Anyway, I will have to see with him in order to find a solution because I
> think that having a (in libtool's vocabulary) current version of 513 isn't
> really acceptable...
To be honest, there's an important factor I left out, which is that
sonames should match other releases (eg. upstream and other distros)
where possible, for cross-release binary compatibility.
You might want to have a look at the debian-mentors archives, too. I believe
this sort of thing gets discussed there on occasion, in more detail that I've
done.
--
-----------------------------------------------------------
Paul "TBBle" Hampson, MCSE
8th year CompSci/Asian Studies student, ANU
The Boss, Bubblesworth Pty Ltd (ABN: 51 095 284 361)
Paul.Hampson@Anu.edu.au
"No survivors? Then where do the stories come from I wonder?"
-- Capt. Jack Sparrow, "Pirates of the Caribbean"
This email is licensed to the recipient for non-commercial
use, duplication and distribution.
-----------------------------------------------------------
| |
| Alexis Papadopoulos 2005-07-07, 7:56 am |
|
>It's a single headache for the one library developer/packager, as
>opposed to headaches for _every user_ of the library.
>
>
Yes indeed, but it's still a headache for one person ;).
>You might want to have a look at the debian-mentors archives, too. I believe
>this sort of thing gets discussed there on occasion, in more detail that I've
>done.
>
>
I will take a look into debian-mentors, but I've just talked to the
upstream author and can now explain the reason of his choice.
The thing is that the library is written in C++ and makes heavily use of
templates which means that even a small change in the code, that doesn't
change the ABI, might lead to incompatibility.
We therefore checked the existant libraries coded in C++, using
templates and present in the debian distribution. We only used two
examples, libginac and libboost-python. We looked both of the shared
libraries present :
libginac-1.3.so.0.0.0 with SONAME libginac-1.3.so.0 : the version is
contained in the SONAME which means that if a program is linked against
libginac-1.3 and only libginac-1.4 is present on the system, it will fail.
Same goes for libboost-python : libboost_python-gcc-mt-1_32.so.1.32.0
with SONAME libboost_python-gcc-mt-1_32.so.1.32.0
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Stephen Frost 2005-07-07, 5:58 pm |
| * Alexis Papadopoulos (Alexis.Papadopoulos@imag.fr) wrote:
> Yes indeed, but it's still a headache for one person ;).
If that one person isn't willing to deal with it then that person
shouldn't be writing libraries. 
> I will take a look into debian-mentors, but I've just talked to the
> upstream author and can now explain the reason of his choice.
Unfortunately that doesn't make his reasoning right.
> The thing is that the library is written in C++ and makes heavily use of
> templates which means that even a small change in the code, that doesn't
> change the ABI, might lead to incompatibility.
There's no 'might' about it... Either it changes the ABI or it doesn't.
ABI does mean more than just symbols though and so, yes, you do have to
be careful and realize when you make an ABI change.
> We therefore checked the existant libraries coded in C++, using
> templates and present in the debian distribution. We only used two
> examples, libginac and libboost-python. We looked both of the shared
> libraries present :
> libginac-1.3.so.0.0.0 with SONAME libginac-1.3.so.0 : the version is
> contained in the SONAME which means that if a program is linked against
> libginac-1.3 and only libginac-1.4 is present on the system, it will fail.
Generally this is done when there's an API change (which also implies an
ABI change) that is pretty significant such that programs written to work
with the old API will have to be updated/significantly modified.
The expectation is that there would be, in the above case, point
releases such as '1.3.1' which would be API-compatible, and if
ABI-compatible then the SONAME wouldn't change, and if not
ABI-compatible then it would change.
Thanks,
Stephen
| |
| Alexis Papadopoulos 2005-07-07, 5:58 pm |
|
>If that one person isn't willing to deal with it then that person
>shouldn't be writing libraries.
>
>
>
Never said that...
>
>Unfortunately that doesn't make his reasoning right.
>
>
>
Nor that. I merely meant that I understood why he chose this kind of SONAME.
>
>There's no 'might' about it... Either it changes the ABI or it doesn't.
>ABI does mean more than just symbols though and so, yes, you do have to
>be careful and realize when you make an ABI change.
>
>
>
The thing is that every change in a template class or function in the
shared library will lead to an ABI change (except some rare cases).
Since the majority of the modifications are made in this section of the
library I don't find absurd to modify the SONAME on each new compilation
of the library (only of course if modification has been made since last
compilation).
This goal of all this is to make the update of SONAME as far as I can
automatically.
The idea now is to keep two records, one with the sources version and
one with the current soname. On each modification (and CVS commit) of
the sources, if changes have been made to the library's source files,
SONAME is increased, sources version is always incremented. The SONAME
will be (as inspired from libginac) in the following form :
lib*-X.Y.so.0.0.0 (the ending 0.0.0 because of libtool's usage)
or maybe lib*.so.XY (format that is used today by the upstream author)
Alexis Papadopoulos
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Bob Proulx 2005-07-07, 5:58 pm |
| Alexis Papadopoulos wrote:
> I'm actually making some .deb packages out of a single source archive.
> One of them should contain a shared library. The upstream author's
> package's version is 5.13 and the soname of his library has been set to
> 513. After having contacted him, he told me that was done because
> apparently each time the new version of the library became uncompatible
> with the previous ones, the major version should be incremented,
> something that was disturbing because the library is often subject to
> modifications leading to incompatibilities.
Is this shared library actually used by other programs? Or only
within the internal use of this particular project? If the latter
then I would avoid packaging it as a shared library at all. If the
shared library is not used by other programs then I would covert the
build to link the project library code as an archive library for
Debian. That should avoid most of the problems.
Bob
| |
| Stephen Frost 2005-07-07, 5:58 pm |
| * Alexis Papadopoulos (Alexis.Papadopoulos@imag.fr) wrote:
> The thing is that every change in a template class or function in the
> shared library will lead to an ABI change (except some rare cases).
> Since the majority of the modifications are made in this section of the
> library I don't find absurd to modify the SONAME on each new compilation
> of the library (only of course if modification has been made since last
> compilation).
You're sure the ABI for template classes is that sensitive? If it is
then they probably shouldn't be in a library to begin with..
> This goal of all this is to make the update of SONAME as far as I can
> automatically.
That's almost certainly a terrible idea.
> The idea now is to keep two records, one with the sources version and
> one with the current soname. On each modification (and CVS commit) of
> the sources, if changes have been made to the library's source files,
> SONAME is increased, sources version is always incremented. The SONAME
> will be (as inspired from libginac) in the following form :
> lib*-X.Y.so.0.0.0 (the ending 0.0.0 because of libtool's usage)
> or maybe lib*.so.XY (format that is used today by the upstream author)
The SONAME needs to match across distributions so it really needs to be
managed (and managed correctly) by upstream. If every change to the
library requires an SONAME change then it almost certainly should not
*be* a library. It would be rather disappointing if what you're saying
about C++ template classes is really accurate. Personally, I suspect
it's not.
Thanks,
Stephen
| |
| Alexis.Papadopoulos@imag.fr 2005-07-07, 5:58 pm |
| > Is this shared library actually used by other programs? Or only
> within the internal use of this particular project? If the latter
> then I would avoid packaging it as a shared library at all. If the
> shared library is not used by other programs then I would covert the
> build to link the project library code as an archive library for
> Debian. That should avoid most of the problems.
Well the goal is to make it available to other programs, so a shared library is
preferable. A -dev package will be available too to allow users to compile
binaries and link them against the .so library...
-------------------------------------------------
envoyé via Webmail/IMAG !
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Alexis.Papadopoulos@imag.fr 2005-07-07, 5:58 pm |
| > That's almost certainly a terrible idea.
I somehow expected that might come up. I didn't fell to comfortable with this
idea, but I think there must be another solution than simply doing it "by hand",
a more "elegant" way.
> The SONAME needs to match across distributions so it really needs to be
> managed (and managed correctly) by upstream.
I have access to upstream's CVS so if any modification is to be made, it will
concern the whole project not only the debianization of it.
> If every change to the
> library requires an SONAME change then it almost certainly should not
> *be* a library. It would be rather disappointing if what you're saying
> about C++ template classes is really accurate. Personally, I suspect
> it's not.
That's what the upstream author explained me, and that's what I want to find
out. Two possibilities, either the upstream author has missed something, or
there is a proper way of dealing with this kind of situation.
One example that might fail :
let's say we have a shared library with 2 source files : g.cc and g.h
g.h :
template <class T>
void g (T x);
g.cc :
template <class T>
void g (T x) {
cout << x;
}
The .h file has to include the .cc one in order for the compilation to work.
That leads to a shared library that we'll call libg.so.1.0.0
Let's say now that I compile a program `prgm` and link it against the above library.
Now if I decide to change one line of g.cc :
cout << x;
becomes
cout << x << endl;
and if I don't change the SONAME (the ABI hasn't changed, there doesn't seem to
be a reason to increment the SONAME), and call `prgm`, I want the the newline on
the output, because the code of g isn't in the .so library but in prgm itself.
According to the upstream author, since the template (class T) isn't known, we
cannot insert the code of g into the library (which seems normal).
My programming skills are limited, therefore I'm doing my best to explain
myself, hope it was clear enough...
Alexis Papadopoulos
-------------------------------------------------
envoyé via Webmail/IMAG !
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Alexis.Papadopoulos@imag.fr 2005-07-07, 5:58 pm |
| > That is just plain wrong. :-) With templates, you are supposed to
> provide the template implementation either in the header or in a file
> included by the header (convention is to name them .tcc and place them
> next to the header). The usual rule applies: Everything that does not
> generate code by itself should be in a header file.
Well I did say that : "The .h file has to include the .cc one in order for the
compilation to work."
Now if you decide to leave the code that I put into g.cc only the .h file, it
works too...
> Yes, so the ABI doesn't change in this case.
It doesn't, and the modification isn't very important so it isn't a problem. But
that was only an example, but the body of g can be modified in a way where it
could lead to a failure (because of the use of templates), therefore the SONAME
muste be changed so as to force usage of the new library.
Alexis Papadopoulos
-------------------------------------------------
envoyé via Webmail/IMAG !
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Stephen Frost 2005-07-07, 5:58 pm |
| * Alexis.Papadopoulos@imag.fr (Alexis.Papadopoulos@imag.fr) wrote:
>
> I somehow expected that might come up. I didn't fell to comfortable with this
> idea, but I think there must be another solution than simply doing it "byhand",
> a more "elegant" way.
You can't really automate ABI checking... You can compare symbols and
whatnot, but that's not enough.
> That's what the upstream author explained me, and that's what I want to find
> out. Two possibilities, either the upstream author has missed something, or
> there is a proper way of dealing with this kind of situation.
>
> One example that might fail :
> let's say we have a shared library with 2 source files : g.cc and g.h
>
> g.h :
> template <class T>
> void g (T x);
>
> g.cc :
> template <class T>
> void g (T x) {
> cout << x;
> }
>
> The .h file has to include the .cc one in order for the compilation to work.
Errr... Wait a minute, how's this supposted to work, exactly? I think
we may need to see some real code here. If the .h is including the .cc
then the library doesn't actually need to be linked against anyway...
Let's see a real example where a template class is used and actually
works in a library and then we can talk about it a bit better.
Thanks,
Stephen
| |
| Stephen Frost 2005-07-07, 5:58 pm |
| * Alexis.Papadopoulos@imag.fr (Alexis.Papadopoulos@imag.fr) wrote:
> Well I did say that : "The .h file has to include the .cc one in order for the
> compilation to work."
> Now if you decide to leave the code that I put into g.cc only the .h file, it
> works too...
The template class has to actually be included, and so really becomes
part of the API... I'd be curious if it'd be possible to change the
template class itself in such a way as to make it not be ABI-compatible
with the actual .so...
>
> It doesn't, and the modification isn't very important so it isn't a problem. But
> that was only an example, but the body of g can be modified in a way where it
> could lead to a failure (because of the use of templates), therefore the SONAME
> muste be changed so as to force usage of the new library.
Alright, how about you provide an example of this...?
Thanks,
Stephen
| |
| Anthony DeRobertis 2005-07-07, 5:58 pm |
| Alexis.Papadopoulos@imag.fr wrote:
> g.h :
> template <class T>
> void g (T x);
>
> g.cc :
> template <class T>
> void g (T x) {
> cout << x;
> }
>
> The .h file has to include the .cc one in order for the compilation to work.
> That leads to a shared library that we'll call libg.so.1.0.0
> Let's say now that I compile a program `prgm` and link it against the above library.
There are two issues here. The first is that you are not really linking
against the code in the shared library. GCC does not support template
export (for damn good reasons, really), and certainly does not support
it through shared libraries. In all likelyhood, the compiler is actually
inlining g().
If you were really using the shared library, you would not have to
include the .cc file. You're not. Your .so most likely does not even
export the instantiation of g<T> you need.
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Steve Langasek 2005-07-07, 5:58 pm |
| On Thu, Jul 07, 2005 at 06:45:13PM +0200, Alexis.Papadopoulos@imag.fr wrote:
[vbcol=seagreen]
> Well I did say that : "The .h file has to include the .cc one in order for the
> compilation to work."
> Now if you decide to leave the code that I put into g.cc only the .h file, it
> works too...
[vbcol=seagreen]
> It doesn't, and the modification isn't very important so it isn't a problem. But
> that was only an example, but the body of g can be modified in a way where it
> could lead to a failure (because of the use of templates), therefore the SONAME
> muste be changed so as to force usage of the new library.
The SONAME refers to the ABI of the *shared library*. Any template-related
code that's included in the C++ headers is not, itself, part of the shared
library, so that's no reason to change the SONAME.
--
Steve Langasek
postmodern programmer
| |
| Anthony DeRobertis 2005-07-07, 8:49 pm |
| [I think you meant to sent this to -devel as well as just me
privately... I hope you don't mind me sending my response back to -devel]
Alexis.Papadopoulos@imag.fr wrote:
> But is this possible ? If for instance I know that T will be double, int and a
> "custom" class, can I force the code for g<double>, g<int> and g<custom> to be
> in the shared library ?
I think it is using explicit instantiation. However, you lose a lot of
the advantages of C++ templates --- ability to inline code, and, most
importantly, ability to instantiate with an arbitrary (compatible) type.
If you decide to go this route, though, you will need to get rid of the
..cc files from the -dev package; otherwise, you risk breaking the one
definition rule. (Maybe this is what you meant by every change needing a
new soversion?)
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
| |
| Alexis Papadopoulos 2005-07-08, 2:49 am |
| Hi,
I'm not having elements to reply immediatly and therefore won't be able
to respond right away. I'm going to take a look at the code and
recontact the upstream author in order to have a more precise idea of
the exact problem, since I missed some things apparently...
Thanks for your patience
Alexis Papadopoulos
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
|
|
|
|
|