<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2653.12">
<TITLE>RE: [Lsb-CommonPackaging] Re: extension of lsb packages</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>:-( Lookout seems to be mangling my inlines... Apologies if it's hard to read...</FONT>
</P>
<P><FONT SIZE=2>-----Original Message-----</FONT>
<BR><FONT SIZE=2>From: Theodore Tso [<A HREF="mailto:tytso@mit.edu">mailto:tytso@mit.edu</A>] </FONT>
<BR><FONT SIZE=2>Sent: 06 March 2002 14:37</FONT>
<BR><FONT SIZE=2>To: Anthony W. Youngman</FONT>
<BR><FONT SIZE=2>Cc: 'msw@redhat.com'; lsb-discuss@lists.linuxbase.org; lsb-taskforce1@lists.sourceforge.net; George Kraft IV</FONT>
<BR><FONT SIZE=2>Subject: Re: [Lsb-CommonPackaging] Re: extension of lsb packages</FONT>
</P>
<BR>
<P><FONT SIZE=2>On Wed, Mar 06, 2002 at 08:10:19AM -0000, Anthony W. Youngman wrote:</FONT>
<BR><FONT SIZE=2>> Which is why I'm trying to get a new standard that says "here's an </FONT>
<BR><FONT SIZE=2>> api. With this, ANY packaging program can talk to ANY distro package </FONT>
<BR><FONT SIZE=2>> database and query it".</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Bingo. All packaging problems solved. You can run any installer on top </FONT>
<BR><FONT SIZE=2>> of any distro and check dependencies, log the fact that you've </FONT>
<BR><FONT SIZE=2>> installed stuff, etc etc. The trouble with current install mechanisms </FONT>
<BR><FONT SIZE=2>> is that they are installer, DBMS, packager, and god knows what else </FONT>
<BR><FONT SIZE=2>> all stuffed into one bundle. If we separate the separate functions </FONT>
<BR><FONT SIZE=2>> into separate tools like all good linusers do, then the problem gets a </FONT>
<BR><FONT SIZE=2>> lot simpler.</FONT>
</P>
<P><FONT SIZE=2>If you assume that the packaging system is responsible for installing packages, then it isn't that hard. You just simply have a couple of operations, really:</FONT></P>
<P> <FONT SIZE=2>1) /usr/lsb/lib/install_package <package_file></FONT>
<BR> <FONT SIZE=2>2) /usr/lsb/lib/remove_package package_name [version]</FONT>
<BR> <FONT SIZE=2>3) /usr/lsb/lib/query_package package_name [version]</FONT>
</P>
<P><FONT SIZE=2>Of course, then the only reason why you need a fancy installer program is to allow the user to choose which sub-packages the user wishes.</FONT></P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>Well, I was thinking possibly along those lines...</FONT>
<BR><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>If you're thinking more in the Windows model, where each vendor has their own propietary setup.exe program which is responsible for copying files in, then that becomes a horse of a very different color. You need to realize that (a) the packaging system keeps track of which file is owned by which package, and often keeps a checksum information about files, so it can determine if a package has been corrupted, </FONT></P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>So when the installer tells the package database what it's installed, it has to include a list of files ... fine.</FONT>
<BR><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>(b) it's kinda stupid to have each vendor replicating the code to copy a file into a standard location, and </FONT>
</P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>It's also kinda stupid to have loads of static and/or duplicate code where different lsb packages just happen to share functionality that isn't defined in the lsb spec. It would be nice for eg Kylix to say it had installed all the dev libraries, so when the user installs an app written by someone else, the installer realises it doesn't need to install duplicate copies of all the runtimes... maybe several times over if I install several Kylix-based packages from several different sources.</FONT></P>
<P><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>(c) the API between the installer program and the packaging system gets *very* complicated, because of the need of the installer to report where each file is to be installed first.</FONT></P>
<P><FONT SIZE=2>It also doesn't solve any of the hard problems of packaging, which include:</FONT>
</P>
<P><FONT SIZE=2>1) Packaging namespace. Each distribution assumes that it, and only it, controls the namespace of packages. So they will freely create packages without worrying about collisions.</FONT></P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>Already dealt with. We've dealt with it in lsb by creating an lsb namespace. The lsb imposes a domain-name-style namespace. Don't comply? You're not lsb-compliant!</FONT></P>
<P><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>2) Version numbers. Each distribution uses a versioning scheme which is roughly related to the original upstream author's versioning scheme, but there are both "epoch" numbers which are more significant than the base version, as well as "release" numbers which are less significant than the base version.</FONT></P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>Again, dealt with. We may need exceptions, but in general we ought just to be able to use the original author's numbering scheme. What happens if there are functionality-damaging bugs and the original author doesn't fix it I don't know, but that's hopefully an unlikely occurrence.</FONT></P>
<P><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>3) Incompatible event models for installation and deinstallation and upgrade scripts. The pre- and post- install scripts can be hand-waved away if you assume the propietary, hand-written installation program for each package. (Although this clearly wouldn't scale for the several hundred packages in a base distribution, but we'll ignore that point for now). However, how and when the scripts which get run when the package is deinstalled or upgraded *do* need to be standardized, since the installation program won't be running at that point.</FONT></P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>How does lsb do it now? Same problem, and same solution!</FONT>
<BR><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>4) Detailed configuration information for each base package to be provided by the distribution. The FHS provides general guidance to the distributions, but there are all sorts of details about how Apache is to be configured, or what the precise directory name is used in</FONT></P>
<P><FONT SIZE=2>/var/lib, which do differ from distribution to distribution. </FONT>
</P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>As above. How do we do it now? If my lsb app needs Apache, then my spec suffers exactly the same difficulties as the lsb suffers now. And it will presumably be able to take advantage of exactly the same solution.</FONT></P>
<P><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>5) Knowledge of how to modify system configuration. If a package needs to add a port to /etc/services, how should it do so such that it will survive an update of the base distibution? How about /etc/inetd.conf? Etc.</FONT></P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>And again. As above.</FONT>
<BR><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>All of these combine to make the problem of dependencies very, very, hard. If my program needs to be able to send mail, what package (or virtual package) do I need to list in my dependencies to guarantee that exim or postfix or sendmail will be installed?</FONT></P>
<P><FONT SIZE=2>[awy]...[/awy]</FONT>
</P>
<P><FONT SIZE=2>What package do I need to depend upon to guarantee that Apache will be installed. And if my package is providing an Apache plug-in module, where has the distribution decided to place the standard Apache plug-ins? For that matter, where has the distribution decided to place the Apache configuration files? Or log files?</FONT></P>
<P><FONT SIZE=2>[awy]...[/awy]</FONT>
</P>
<P><FONT SIZE=2>And if my package requires at least Apache 1.3.12, how do I state that? Different distribution may have different epoch numbers, and in some cases, different ways of encoding the version number if the upstream author hasn't been careful about releasing software with consistent version numbers.</FONT></P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>With an lsb-defined namespace, we simply say Apache >= 1.3.12. Simple!</FONT>
<BR><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>So to say, "all we need to do is to specify an API", and like pixie dust, expect that all of the hard problems of packaging will just magically disappear, is a bit naive. </FONT></P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>Yes, I know it is incredibly naive. But I fail to see how, using my solution, it's going to be *harder* to solve than using the current mechanism. The solutions we come up with here will carry over.</FONT></P>
<P><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>The LSB 1.0 has a partial solution to these problems, but we do so by significantly constraining what an LSB-compliant package can do. An LSB-complaint distribution can only depend on a single package, named "lsb". We restrict what sort of things the package is allowed to assume (because they're different on various distributions). We require long package names, such as "lsb-quicken.com-turbotax", which are not universally popular.</FONT></P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>And I've simply sought ways to *extend* that, with the intention of increasing flexibility. Just like with lsb 1 we restrict heavily what the package is allowed to assume, I've done the same, with my "necessary and sufficient" set (which pretty much means "if you require more than current lsb, then you're broken"). But it also allows for the addition of a lot of *OPTIONAL* extras, which allows packagers to innovate, try things out, and then ask for them to be added to the standard.</FONT></P>
<P><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>> Of course, the fly in the ointment is this becomes a "standard </FONT>
<BR><FONT SIZE=2>> creating" exercise, rather than the current "codifying existing </FONT>
<BR><FONT SIZE=2>> standards" exercise and there'll be horrendous politics. But if you </FONT>
<BR><FONT SIZE=2>> think about it, the proposed approach will work perfectly, even with </FONT>
<BR><FONT SIZE=2>> an ISV who wishes to create a totally proprietary, obscured to the </FONT>
<BR><FONT SIZE=2>> hilt, 100% totally self owned code, setup.exe to run on a </FONT>
<BR><FONT SIZE=2>> lsb-compliant system, AND IT WILL WORK. If we can address and allow </FONT>
<BR><FONT SIZE=2>> for that setup, it then makes it easy for every one else to be far </FONT>
<BR><FONT SIZE=2>> more sharing and co-operative.</FONT>
</P>
<P><FONT SIZE=2>Well, ask yourself this. What parts of a propietary, obscured to the hilt, hand-rolled installer would the ISV really want to keep secret? And why? Copying files around? That's not secret. The list of what files are installed? We can compare the system before and after the installation, and know what files where installed/modified.</FONT></P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>That comment was not meant seriously. It's just a genuine worst-case scenario, and as I said, if we can deal with that then it'll be dead easy to deal with developers working in friendly co-opetition.</FONT></P>
<P><FONT SIZE=2>[/awy]</FONT>
</P>
<P><FONT SIZE=2>The only thing which really needs to be kept secret/propietary is the initial configuration of the package, and arguably that shouldn't be part of the installation process in the first place. You *want* to separate that out, and perhaps the most you should do is to be able to call the initial configuration tool from the post-installation script. </FONT></P>
<P><FONT SIZE=2>The bottom line is that I don't think there's any real value in having a windows-style SETUP.EXE custom installer. People may be accustomed to it because it's what Windows used to do, but even in the newer Windows system, Microsoft has actually been moving to the packaging model used by Linux distribution systems, where the vendor provides a package, and the packaging system takes care of installing it. Indeed, with Windows 2000, Microsoft was boasting about how for the first time, they can verify whether or not a package has been corrupted or not --- of course, they conveniently failed to mention that Linux and its packaging systems (especially RPM) has been able to do this for years....</FONT></P>
<P> <FONT SIZE=2>- Ted</FONT>
</P>
<P><FONT SIZE=2>[awy]</FONT>
<BR><FONT SIZE=2>Which is why my spec is based on (a) separating the front and back ends of the installer, (b) abstracting the interface, and (c) trying to spec that interface such that it's compatible with rpm, apt, and lsb.</FONT></P>
<P><FONT SIZE=2>Once we've got that, and a mechanism for extending it, then co-opetition means the whole shebang is going to evolve rapidly.</FONT></P>
<P><FONT SIZE=2>As I've said before, "monolithic" is NOT the linux way. You wouldn't expect a database client to directly modify the files on disk, would you? It goes through a SQL interface or similar and lets a DBMS look after it. Why not apply *exactly* the same approach to an install tool. After all, the front end currently DOES modify the back end database, doesn't it? How stupid is that!</FONT></P>
<P><FONT SIZE=2>At the end of the day, all the objections you've brought up are either (a) obvious, and with obvious solutions, or (b) ones the lsb is already falling over, and we can reuse the same solution.</FONT></P>
<P><FONT SIZE=2>And all I'm trying to do is abstract the interface between the installer and the package database, which means we can work out a common subset between the main packagers, and seek to build on it. At present, I feel all this debate is having a seriously stifling effect on package management development, and if we start codifying things the way they are, things are going to get worse.</FONT></P>
<P><FONT SIZE=2>I'm quite happy with the way things are going for lsb 1. We need the current standard. But things need a total rethink for lsb 2. When this came up before I got the impression Jeff and Wichart grok'd what I was getting at. Albert most certainly did. And I would not be in the slightest bit surprised if my ideas didn't get taken up in some form or other. I'm a database professional. I make my living programming the damn things. And the way the package databases are managed just feels horribly "inelegant". As you ought to know, "if it's inelegant, it's wrong!". Maybe I'm wrong, but we are talking about my personal sphere of expertise...</FONT></P>
<P><FONT SIZE=2>Cheers,</FONT>
<BR><FONT SIZE=2>Wol</FONT>
</P>
<BR>
<P><B><FONT SIZE=2>This transmission is intended for the named recipient only. It may contain private and confidential information. If this has come to you in error you must not act on anything disclosed in it, nor must you copy it, modify it, disseminate it in any way, or show it to anyone. Please e-mail the sender to inform us of the transmission error or telephone ECA International immediately and delete the e-mail from your information system.</FONT></B></P>
<P><B><FONT SIZE=2>Telephone numbers for ECA International offices are: Sydney +61 (0)2 9911 7799, Hong Kong + 852 2121 2388, London +44 (0)20 7351 5000 and New York +1 212 582 2333.</FONT></B></P>
</BODY>
</HTML>