Build scripts are not enough to create a package. There are some other necessary informations used in the build phase and also some meta data that is needed to be placed in the output package. We decided to separate the meta data from the build scripts and placed them into a file called pspec.xml. Pspec.xml holds build related informations like runtime and buildtime dependencies of a package, path and sha1sum of the source archive, patches to be applied and other meta data informations like packager information, description, summary, homepage of the software project and changelog info of the package, etc. We decided to put the data into a XML file instead of using an ad-hoc text format. This makes it easier to process and to write independent tools for various purposes. It is also much more structured and readable.
Following is the pspec.xml file of joe source package as an example:
<?xml version="1.0" ?>
<!DOCTYPE PISI SYSTEM "http://www.pardus.org.tr/projeler/pisi/pisi-spec.dtd
<Summary>Joe text editor.</Summary>
<Summary xml:lang="tr">Joe metin dzenleyicisi</Summary>
<Description>Joe is a small text editor with sytax highlighting.</Description>
<Archive sha1sum="aeee6bb71ed1e9b9c9a2a64d8e078a06245957e5" type="targz">
One other virtue of using XML format is that it is possible to easily divide packages by defining more Package tags into a pspec.xml file.Components and Categories
PiSi defines two relations to identify a package: Components and Categories. We can say a x package is part of y component and that x package belongs to z category. With components we have a ``partof'' relation and with categories we have an ``isa'' relation.
Most of the time the directory hierarchy of the source package also gives us the component of the package, except for some rare corner cases. On the other hand, category of the package is explicitly defined in the pspec.xml file with an IsA tag.
We can use the previous given example, joe package to show these relations. Source package joe is under applications/editors/joe directory in our source repository. There we can see that the component of the joe package is ``applications.editors''. The category of the package is defined in the pspec.xml as ``app:console''.
We can use the components to create a minimal system with all the packages under those components installed. A component is an installable entity for PiSi. You can install the ``system.base'' component to have a minimal base system and the ``desktop.kde.base'' component to have a minimal desktop environment.
Currently the category info is used by Package Manager that has an option to show only the packages that belong to ``app:gui'' category.What PiSi Brings
As in all advanced package systems, PiSi has its own dependency resolver. It is designed to be within PiSi from the beginning and so different from DPKG and RPM, not done with wrapper tools. It is based on a topological sort algorithm. Kept as simple as possible to be able to work as fast and efficient as possible. It is adequate for our needs and we do not plan to create a beast out of it. It currently allows for the build system to provide the dependent and conflicting packages with relative versioning informations.Scenarios
One of the difficult parts of developing a package system is to reproduce the reported bugs to figure out what went wrong. But it is not always easy to reproduce the problems. Or sometimes you figure out a possible scenario that resulted as the reported bug, but you have to make sure that it is that case. And to make sure you fixed it, you have to have a way to reproduce the bug easily to test your fixes. PiSi ScenarioAPI is just for that purpose. By using the ScenarioAPI you write scenario scripts that ends as the bugs reported. So by writing scenario scripts, you have easily reproducable cases. You can add many packages as you like for the test cases. The packages will automatically be created on runtime of the script.
ScenarioAPI uses PiSi's ability of working with a different system root. Every command may take an optional -D parameter that points to a destination directory that resembles a chroot system for PiSi. The directory may be an empty one. When PiSi sees there is no database environment in the directory, it creates one to work with. So the scenario scripts run in an isolated space. Image creation tools also uses this feature.
Following is a sample scenario script taken from the PiSi trunk that tests when the conflicting package removals happen. According to the bug report, removal happens before any package is fetched. But it should be done after fetching all the packages and before starting to upgrade to the fetched packages.
from pisi.scenarioapi.scenario import *
XORG = "xorg"
QT = "qt"
XORG_SERVER = "xorg-server"
XORG_VIDEO = "xorg-video"
XORG_FONT = "xorg-font"
repo_added_package(XORG_SERVER, with_conflicts(XORG), with_dependencies(XORG_VIDEO, XORG_FONT))
repo_version_bumped(QT, with_removed_dependencies(XORG), with_added_dependencies(XORG_SERVER))
Source and Binary Repositories
PiSi supports both source and binary repositories.
The source packages of Pardus are developed at our svn repositories. The svn repositories are also used as source repositories.
The binary packages are kept at our binary repositories.  When new package releases are committed; the Buildfarm creates binary packages from source repositories and moves them to the corresponding binary repositories. Buildfarm package queue is managed manually by repository administrators.
The source and binary repositories are represented by their index files named pisi-index.xml. Both created by PiSi's index command executed from the root of the repository. The index command creates a pisi-index.xml file out of that repository tree. Adding and using new repositories to PiSi is as simple as:
pisi add-repo contrib http://paketler.pardus.org.tr/contrib/pisi-index.xml.bz2
pisi add-repo pardus-2007-sources http://svn.pardus.org.tr/pardus/2007/pisi-index.xml.bz2
Building and Emerging of Packages
In the example above, we have added a source repository named pardus-2007-sources. The given repository names are arbitrary.
After adding a source repository we have many options to work with the source packages. First of all, since we have all the source package informations from a repository, we can simply build any package by giving only the name of the package as in:
# pisi build kiki
The build command will fetch the original source archive, the patches provided by the source package, additional files and COMAR scripts to a temporary location. If there are missing build dependencies to satisfy, PiSi will try to install them. After building, the binary kiki-x-y-z.pisi package will be created.
The build process is divided into multiple steps. For example, you may not want to build binary package but just to see the source code of the software provided with that package. Or you may want to work on that code and try multiple builds until you are finished and then continue to other steps of the binary package creation operation. The build command steps are: fetch, unpack, setup, build, install, package. Fetch step just fetches the source tarball of the package. Unpack step extracts the archive and then applies any existing patches. Setup, build and install are the actual building steps. The last step, package, is the one that creates the binary ''.pisi'' package. As an example to unpack a package, you do:
# pisi build --unpack kiki
The source packages are extracted and compiled under /var/pisi directory. The output binary package will be written to the current working directory.
The good thing is that you don't need a source repository to work with the source packages. It is possible to build a package by providing only the local path or the remote url of a spec file without having any source repository added to PiSi database.
Here is an example to build the same package by providing only its remote url:
# pisi build http://svn.pardus.org.tr/pardus/devel/applications/games/kiki/pspec.xml
One other feature of PiSi is its ability to emerge source packages from the source repositories. By emerging, PiSi automatically downloads the buildtime and runtime dependencies of the package. If they do not exist in any binary repository, PiSi also tries to emerge those missing dependencies from the source repositories. PiSi does not need the whole source repository to be checked out or downloaded. Because a source repository is represented by the index file that also contains all the dependency informations and building a package by its remote url is possible, for emerging a package, you only need to add the source repository of the package to PiSi database.
Here is an example to emerge a package:
# pisi emerge kiki
LZMA Compressed Packages
LZMA, short for Lempel-Ziv-Markov chain-Algorithm, is a data compression algorithm. We decided to use LZMA compression in our package format for its greater compression ratio than gzip and bzip2. It is about %30 better than gzip and %15 better than bzip2. 
To give a more concrete example: Pardus 1.0 stable release was using gzip compression in PiSi packages. With all our efforts to put all the necessary packages to satisfy the needs of a desktop user, we were only able to put a total of 335 packages. But now in Pardus 2007, by using LZMA compression within PiSi package format, there comes 641 packages with the installation CD. When all the packages that are in Pardus 1.0 CD are extracted to the destination system, it takes 1.891 GB of disk space and that is 3.043 GB of disk space for Pardus 2007.
The package format is a zip archive. This makes it possible to reach the metadata and COMAR files of the package faster by using the standard tools. Under the zip archive, there exists install.tar.lzma file that contains the actual files of the package to be extracted to the system. A sample pisi package content is as following:
Length Date Time Name
-------- ---- ---- ----
1357 10-22-06 21:16 comar/service.py
6824 11-10-06 11:59 metadata.xml
7803 11-10-06 11:59 files.xml
388881 11-10-06 11:59 install.tar.lzma
-------- ------- 404865 4 files
PiSi package comes with a COMAR management script manager.py that is registered to COMAR's database when PiSi's own package is installed. By using this script's services over COMAR, it is possible to do root privileged package operations. Package Manager, the PiSi frontend uses these services to install, remove, upgrade packages or add/remove new repositories to PiSi database. The authorization is done by the COMAR daemon. By extending these services, it will also be possible to remotely manage the package system of any machine.
The xml spec file format makes it possible to provide translations of the description and summary of the packages to other languages. We are aware of the problem that just updating a translation itself would need a release increase and therefore a package update. i18n support is important so we will try to balance that problem by waiting the next release of the package with an actual update need. We are planning to create a delta pisi package format in the future that can also help with this problem. We decided to keep all the related info in the corresponding packages for its ease of maintenance purposes. Description and summary tags of the packages are exported to a standard pot file from the repository and accommodated to our Pardus translation statistics page. Translated po files then are merged with the corresponding source packages in the repository. These operations are done with our repository translation scripts. 4 Near Future
One of the important things PiSi lacks right now is package signing and validation. We are currently working on this and expect to have an implementation soon.
One other feature we are going to work on is delta pisi package format. We decided not to use a binary diff tool like xdelta. LZMA does its job well on compression part so we are working on the possibility of creating packages only with affected or changed files.
There are suboptimal usages of Berkeley DB. PiSi should have been faster in some certain database operations. This is also planned to be fixed in the next major release.
A sandbox support for PiSi is also planned. There are two ways to sandbox a build script within user context: Overriding glibc functions with LD_PRELOAD or intercepting system calls with ptrace. We decided to go with the ptrace, cause former makes it cumbersome to pass Python variables between builder and build scripts. We considered Subterfugue, but although it is extremely flexible with the Python system call hooks, it isn't fast enough for production use in a build farm. Thus we wrote catbox, a small sandboxing module for PiSi. It is completely written in C, and designed to do just required features for package management. Not all features are completed yet, but basic sandboxing works and will be incorporated with PiSi in the future.5 Conclusion
One of the design goals of PiSi was easy development and maintenance of the packages. We think that PiSi had reached that goal in its early releases that made us possible to create that many packages in such a short time with so far no problems.
Pardus tries to stand out with its new and brave approaches in the problematic areas of Linux distributions. With PiSi and COMAR architecture we created a system that now stands on its feet. The next step is to evolve the system with so many things that are possible to do with the provided infrastructure and to gain new developers by capturing their attention with what we see in Pardus.