Pakman User's Guide

Contents

Introduction

This first section provides a gentle introduction to Pakman by walking through an example of using Pakman to retrieve and build a product that consists of several independent components. This is a real working example, so you may follow along, if you like, replicating these steps on your machine.

Setup

Pakman is contained in a single binary executable.

You can copy the executable into a directory that is in your path, or you can add the builds/WinNT directory to your workspace and include that in your path. You can try typing pakman help to verify that you have it installed. See Troubleshooting, below, if you experience problems.

This document describes pakman version 0.996.

Getting a Project

Pakman's “get” sub-command retrieves a package and prepares it for building. Pakman uses a URI-like format to describe package locations. This identifies the type of Version Control System (VCS), the server, and the location within the server.

If you are starting with an empty client workspace (no views mapped), then you should be able to type the following command and see similar output:

$ pakman get p4://acme123/depot/users/bhk/proto/pakman/pak@939260
Getting p4://acme123/depot/users/bhk/proto/pakman/pak@939260
mapping //depot/users/bhk/proto/pakman
mapping //depot/users/bhk/opensource/lua/lpeg-0.9
mapping //depot/users/bhk/proto/runlua
mapping //depot/users/bhk/proto/make.d
mapping //main/build/tools
mapping //make.d
mapping //depot/users/bhk/builds/pkgs/lua-5.1.4
mapping //depot/users/bhk/proto/simp4
mapping //depot/users/bhk/proto/luau
mapping //depot/users/bhk/opensource/lua/luafilesystem-1.4.2
updating client pakmantest
Client pakmantest saved.
syncing p4://acme123.com:1666/build/tools/Darwin/...@939260
 [10 changes]
syncing p4://acme123.com:1666/make.d/...@939260
 [113 changes]
syncing p4://acme123.com:1666/depot/users/bhk/builds/pkgs/lua-5.1.4/darwin_...@939260
 [4 changes]
syncing p4://acme123.com:1666/depot/users/bhk/builds/pkgs/lua-5.1.4/inc/...@939260
 [4 changes]
syncing p4://acme123.com:1666/depot/users/bhk/opensource/lua/lpeg-0.9/...@939260
 [7 changes]
syncing p4://acme123.com:1666/depot/users/bhk/opensource/lua/luafilesystem-1.4.2/...@939260
 [26 changes]
syncing p4://acme123.com:1666/depot/users/bhk/proto/luau/...@939260
 [29 changes]
syncing p4://acme123.com:1666/depot/users/bhk/proto/make.d/...@939260
 [4 changes]
syncing p4://acme123.com:1666/depot/users/bhk/proto/pakman/...@939260
 [20 changes]
syncing p4://acme123.com:1666/depot/users/bhk/proto/runlua/...@939260
 [13 changes]
syncing p4://acme123.com:1666/depot/users/bhk/proto/simp4/...@939260
 [5 changes]
writing /Users/bhk/p4/pkg/pakman/x.min
writing /Users/bhk/p4/pkg/pakman/x.mak
writing /Users/bhk/p4/pkg/runlua/x.min
writing /Users/bhk/p4/pkg/runlua/x.mak
writing /Users/bhk/p4/pkg/make.d/x.min
writing /Users/bhk/p4/pkg/simp4/x.min
writing /Users/bhk/p4/pkg/simp4/x.mak
writing /Users/bhk/p4/pkg/luau/x.min
writing /Users/bhk/p4/pkg/luau/x.mak
Done.  10 packages retrieved.
To build:
    cd pkg/pakman
    make -f x.mak            # builds the package and its dependencies
    make all                 # builds just the package

We can see from the output that Pakman retrieved not just one directory of files, but several. This is because the requested package has dependencies — other packages that must be retrieved in order to enable it to be built. These are described in package description files that Pakman reads from the server. Pakman recursively visits the entire tree of dependencies and retrieves their files.

The following diagram summarizes the packages that make up the project and their relationships to each other:

The pakman package depends directly on five packages and indirectly on four others. The runlua package is required by three other packages — luau, simp4, and pakman — and in turn it directly depends on two other packages. You can generate a diagram like this for any project using the pakman describe command.

When Pakman retrieves sources it does so using “p4 sync”, leaving them under version control. Pakman will automatically add rules to a workspace view (as reported above in the output log) but it maintains a usable workspace: files map simply and directly to repository locations. Pakman never removes files from a workspace view. Developers retain control over their workspace view and can continue to interact with Perforce as usual, editing and submitting changes and resolving conflicts.

Building a Project

Now that the source tree has been constructed, you can issue the command to build the project (as described by Pakman in the above output):

$ cd pkg/pakman
$ make -f x.mak
making ../runlua
making ../luau
making ../simp4
making .

The “x.mak” file knows how to build both Pakman and all of its dependencies, and build them in the correct order. Not all of the dependencies appear in the above output because some dependencies have no build step of their own.

This step does not directly involve Pakman, but it makes use of glue files, like x.mak, that were created by Pakman when the project was retrieved. This means that a Pakman-constructed source tree can be built offline, or without access to a Pakman executable. Glue files use relative paths to refers to other packages, so Pakman source trees are copy-deployable: the tree will be build-able after being copied to another location or another machine.

Further Reading

If you have experience working with Perforce and managing your local workspace views, you might want to read the section on Workspace Management to better understand how it will fit into your working style.

If you want to create new packages, or create or maintain packages, then you will want to read the sections on Pakman Concepts and package.glue.

If you are managing a large project and want to establish efficient processes for a team, then you will be interested mostly in the section on Use Cases and Strategies.

Command Syntax

pakman get

pakman get project/pak                    
pakman get .                              

pakman get p4://p4d/depot/proj/pak        
pakman get p4://p4d:1666/depot/proj/pak   

pakman get //depot/proj/pak               
pakman get p4:///depot/proj/pak           

pakman map

pakman visit

pakman make

pakman describe

pakman help

pakman version

Options

Pakman Concepts

Component-based Builds

Pakman supports component-based builds. The central idea is that a software build process can be decomposed into a set of components. Each component has a collection of source files. A component has its own build step, and a set of dependencies on other components. A component can produce build results: the files that are available after its build step is complete. A component's results are constructed from its own source files and its dependencies. In a large build system, higher-level components can construct their results using the results of lower-level components.

The term “component” implies the ability to substitute one implementation for another. A component has an external interface or contract independent of its implementation. In the case of a build system component, its interface consists of its build results: what files are expected to be there, and what interfaces or contracts those files are expected to meet.

For example, a version 1.1 branch of a component could be substituted for a version 1.0 branch as long as the version 1.1 branch produces all of the result files expected by the 1.0 consumer and as long as those files meet the same functional requirements. Another example of interchangeability is when a pre-built instance of a component is substituted for one that builds from source. In this case the result files are identical although the build step differs.

Each component can be built independently of its “parents” (other components that consume its results). In fact, in Pakman's implementation, build components do not even know of the existence of their parents. Each component is self-contained. This ensures that components can be re-used in other projects or configurations.

One type of component is a component without a build step. In this case, the “results” of such a component consist of the versioned files associated with the component, or some subset of them.

Another common case is a component that supports variants: different sets of build results generated from the same set of sources. A common example is when a library may be compiled in debug or release mode. The concept of variant builds can also apply to build results that do not resemble each other. In addition to debug and release variants, a directory of C source files might be able to generate documentation files or metrics describing code coverage.

Pakman uses the term package generally to refer to a build component. When a component can produce multiple build results, a package refers to a specific variant. Naming a package, thereby, names a specific set of build results.

Theory of Operation

Pakman's main function is retrieval of package trees from version control onto the local file system in a form ready to be built. This process consists of the following steps:

1. Visit: retrieve and process the description file (“pakfile”) for a specified package, then its dependencies, and so forth until the entire package tree has been visited. (More precisely, this is a directed acyclic graph (DAG), not a tree, since diamond-shaped dependencies are allowed.) Version Conflicts are detected at this time.

2. Map: locate or create mappings for each package root directory. Mapping conflicts are detected at this time. See Workspace Management for more information.

3. Sync: retrieve source files from the repository to the local file system. The source files retrieved are those under the root directories for the packages. The package.files parameter may limit the number of files to be retrieved in this step.

4. Glue: write glue files into the local file tree. Glue files are described in the package.glue section.

Reading From VCS or the Local File System

When Pakman retrieves files with “p4:” URIs during the visit stage, it does not “check out” or “sync” the package description files into a local workspace; it reads the contents directly from the VCS server.

If the current workspace contains a local copy of the “p4:” file, the local copy will be used if it has been edited — specifically, when the “read-only” bit is not set in Windows, or the writable permission bit is set in UNIX-derived OSes. The local file will also be used when no copy exists in the server, and directories that exist locally will be recognized even if they do not exist on the server.

Version Conflicts

Perforce does not allow one place in the repository to be synced to two different places in one workspace. When two different packages require different revisions of the same repository file, Pakman will detect this conflict and abandon the “get” operation before syncing or generating any files. These kinds of conflicts can arise when two different versions of a package appear in a dependency graph. For example, if package A requires B@2 and C@2, but B@2 requires C@1, then we have a version conflict.

Package Locations

Pakman identifies packages using a URI-like syntax. A package location consists of the following parts:

There is always a path part (even if only an empty string), but the other parts may be present in the URI or absent. Their presence is signified by a distinctive punctuation character or sequence.

The scheme name “p4” identifies Perforce as the version control system. This is currently the only supported VCS. The scheme “file” is used to identify packages in the local file system, not under version control. Local packages (file:) require no “mapping” or “syncing” steps, and version specifiers are ignored when dealing with local packages.

Locations maybe absolute or relative. Relative locations specify a location relative to a base location that provides context for “resolving” the relative location to an absolute one. If they omit the scheme or host fields, the values from the base are carried over. If the path does not include an initial “/”, then it is treated as relative to the path of the base URI. Elements named “..” denote a parent directory.

Pakman's handing of relative locations is slightly different from the handling of URIs in web browsers. The version field — “@xxx” — is always inherited from the base unless the location being resolved contains a version field of its own. So even when a location looks like an absolute location — e.g. p4://host.domain.com/package/a — it may still inherit a value from the base. This rule is the mechanism by which dependencies inherit the versions of the packages that include them.

Query and Fragment

In standard URI syntax, a query consists of a number of fields delimited by “&” or “;”. Pakman uses the query part to convey package parameters, which are exposed to pakfiles via the package.params member. Each query field can hold a name and value separated by “=”, or simply a value (a “positional” parameter). The ordering of fields that assign names is not significant, and if a name appears more than once the last occurrence overrides earlier ones. For example, “?b=3;b=2;a=1;x” is equivalent to “?x;a=1;b=2”.

In traditional URI/URL usage in web technologies, the query and the fragment play different roles:

• The query is sent to the server. The server assigns meaning to the query. Changing the query can drastically affect the contents of the entity being retrieved.

• The fragment is not sent to the server. It identifies a location within the retrieved entity.

The usage in Pakman is analogous:

• Parameters are provided to the pakfile, which decides what the parameters mean and what build results they produce. Changing a parameter may yield entirely different build results.

• Fragments in Pakman identify a file system location within a build result. For any package URI <pkg>, the result of a dependency named <pkg>#<dir> is equivalent to the <dir> subdirectory within the result of the dependency named <pkg>.

Types of Packages

Pakman deals with two types of packages: described packages and plain packages.

• When a package location names a file that exists in the VCS repository, that file is treated as a pakfile. Pakfiles describe the package properties. In this case, Pakman will retrieve the package described by that file, not just the file itself.

• When a package location's path ends in “/...” it identifies a plain package. Plain packages have no pakfiles, no build step, and no dependencies. All files underneath the directory are retrieved, and those files constitute the build “results” for that package.

When a path does not end in “/...” and does not identify a file then Pakman will take the following additional steps:

1. If the path identifies a directory that contains a file named “pak”, then “pak” is used as the pakfile.

2. If the path identifies a directory that does not contain a file named “pak”, the directory is treated as a plain package.

These additional steps require more Perforce transactions, and are provided for compatibility with older versions of Pakman. In order to make your pakfiles more efficient, specify explicit package locations. Use “<dir>/...” for plain packages, and “<dir>/<pakfile>” for described packages. Future versions of Pakman may display warnings when imprecise names are used.

Params passed to a plain package have no effect. Fragments, on the other hand, may be used with plain packages. When used with a plain package, the fragment will be used to narrow the set of files retrieved from the server. In other words,p4://server/depot/a/...#b/c/ is equivalent to the package file:

root = "p4://server/depot/a"
files = { "b/c/..." }
result = "b/c"

This can be convenient when identifying directories or files within a large repository tree that is not factored as components. The path of the URI specifies the root of the package. In order to avoid Nested Roots, packages should agree on the root directories to be used. The fragment specifies what subset of the tree needs to be retrieved.

Workspace Management

Perforce manages a view for each workspace that maps files in the repository to locations in the local file system. The view does not describe what has been retrieved; it only specifies where things will go once they are retrieved.

Ordinarily, a user would manually construct and validate these workspace views (and perhaps revisit them and inspect and debug them when build problems are suspected). Pakman automates workspace construction, but does so in a way that leaves the user in control: it leaves pre-existing mappings unchanged, and it does not add lines to the client spec that override or mask pre-existing lines.

Before retrieving a package, Pakman ensures that the package's root directory tree is mapped in the view. Due to the way Perforce views are specified, there are three cases Pakman handles differently:

1. The directory is already mapped by the current view. All files that might exist under the directory are mapped locally in a way that matches the directory structure in the repository.

2. The directory is completely absent from the current view. No files that might exist in the package are mapped.

3. The directory is partially or inconsistently mapped. This will be the case when the view includes a subset of the directory tree, or excludes a subset, or maps different parts to different locations.

If the package is already mapped, Pakman will be satisfied with that mapping and continue on. Pakman generally does not care where packages land — only that they are completely and reliably mapped. Likewise, packages themselves should not care where they land, as discussed below under Location Independence.

If the package is completely absent, Pakman will create a map specifically for that package and add it to the workspace view. More on this below.

If the package is partially or inconsistently mapped, Pakman will exit with a message that it cannot map the package . Pakman will not attempt to automatically remedy this situation because that would interfere with existing maps. Instead it exits with an error message, describing how the view can be edited to create a complete mapping for the directory. However, as long as the root directory is mapped, then the user may invoke Pakman with the --force option to cause Pakman to proceed anyway, even where parts of the source tree are excluded or misplaced. This is a way for users to indicate that workspace mappings, while they do not completely and accurately reflect the package descriptions, are what the user intends to use.

Creating Maps

When Pakman creates a mapping for a previously unmapped package, it uses a simple default rule to choose the local workspace location, placing all packages under a directory called “pakman” underneath the workspace root. Pakfiles can specify different rules as described in package.mapping.

Pakman will not use a local workspace location that conflicts with existing maps. If the local workspace location is already used by existing maps in the workspace, the Pakman will try alternatives. If none of its alternatives are available, Pakman will exit with an error message asking the user to manually map the package or make the default location available.

Editing Workspace Views

After Pakman has modified a view in order to retrieve a package, users can feel free to edit their workspace views. Pakman does not expect any its additions to remain unchanged forever. After modifying a workspace view, be sure to run Pakman again so it can re-validate the view, potentially add new maps, re-sync needed packages, and re-generate glue files to reflect the changes.

Overlay Mappings

Perforce overlay mappings (e.g. +//depot/a1/... //client/a/...) are ignored by Pakman. This allows “sparse branches” to be done on subsets of packages mapped by Pakman, although the user must take care to manually sync the overlay after using Pakman to sync the package.

Nested Roots

When the root directory of one package lies under the root directory of another package, this is called a nested root. Nested roots should be avoided, since they are prone to creating mapping conflicts. If Pakman creates a map when retrieving the “inner” package, then the workspace will have a partial mapping for the “outer” package, and this will make Pakman unable to automatically create a mapping for the “outer” package at some later time.

When nested roots are encountered within a single 'get' command, Pakman will print a warning. It will also sidestep the conflict by creating the mapping for the higher-level directory. However, Pakman will not be able to detect the condition or sidestep the conflict if the two packages are used in different package trees and happen to be retrieved in different invocations of 'pakman get' on the same machine.

Pakfiles

A pakfile describes a software component: what its dependencies are, how to configure it to locate its dependencies, and how to build it. These files are typically named pak or end in .pak. Each pakfile contains Lua 5.1 source code that acts as a constructor for a package object. Here is an example pakfile:

-- pakfile for projects/baz
deps = {
   INCDIR = "/common/inc",
   TOOLDIR = "/tools/WinNT"
}
glue = { "x.min", "x.mak" }
commands.make = "make"

In this example, deps and glue are assigned, and the make field of the commands table is assigned. These variables look like global variables in Lua, but Pakman executes the constructor in a restricted environment so that the values assigned are captured as properties of a package.

Instead of assigning variables in its environment, as in the above examples, a pakfile may return a table that describes the package. In that case, the variables in the environment are ignored. Default values will be used for any property that is nil in the returned table.

The following table summarizes the properties that may be assigned or modified by the pakfile, and the values to which they are initialized before the pakfile is executed:

All relative URIs or file paths are treated as relative to the root URI or directory. When root itself is relative, it is treated as relative to the pakfile.

Note that a package that uses the default values for all values behaves equivalently to a plain package URI, such as “/tools/WinNT/...”. Plain packages sync the entire sub-tree, have no dependencies, no build command, and require no generated files.

The different package properties that may be assigned by a package file are described below.

package.commands

Each package can specify the commands that must be issued in order to make or clean it. These commands enable the generation of glue MAK files.

Pakfiles can specify these by assigning fields of the commands property, as in:

commands.make = "make all"
commands.clean = "make clean"
commands.maketree = "make tree"

The commands property's initial value is an empty table, so pakfiles do not have to assign commands itself.

commands.make is the command to make the package, assuming all its dependencies have been built. This command must not attempt to build its dependencies using the glue MAK file, or else infinite recursion will result.

By default, each package has no “make” command, which means that no build step is required to produce its outputs.

commands.clean is the command to clean the package. The “clean” command is unused when there is no “make” command. If there is a make command and no clean command, commands.clean defaults to commands.make with a space and the word “clean” appended to it.

commands.maketree is the command to build the package and all of its dependencies. If this is not provided by the pakfile but a MAK glue file is generated, pakman will provide a default. Unlike the make and clean values, maketree does not influence generated MAK files; it is only for display to the user in the default message (see package.message).

These values will be processed as described in String Expansion, so they may be assigned as functions or strings with variable substitutions.

package.deps

The deps property is used to express dependencies on other packages. A package cannot be built until its dependencies are first built. When one package, say package A, lists another, say package B, as a dependency, Pakman will retrieve package B when it retrieves package A, and it will generate makefiles that build package B before package A.

Take the following pakfile excerpt for example:

deps = {
   B_DIR = "/packages/b",
   C_DIR = "/packages/c",
}

The meaning of the above can be stated in English as:

• When retrieving package A, packages B and C must also be retrieved.

• Before building package A, packages B and C must first be built.

Dependencies are specified as a Lua table. For those new to Lua, a table is a data type that maps values to other values. It is similar to what Perl, Python, and Ruby call a “hash” and what JavaScript calls an “object.” Each dependency has a variable name (table key) and a package location (table value).

Every dependency must have a unique name. If you use the same name twice, the last assignment will override the first. The dependency name will appear in generated “min” glue files as the name of a variable that holds the dependency's result directory.

The values associated with the names are package locations, either in table form or string form. See Package Locations for information on Pakman's package location strings, and see pmlib for information on the table representation of URIs.

Dependencies are unordered. The order in which they appear in the pakfiles does not specify to the order in which they are retrieved. Pakman's conflict detection ensures that the order in which they are retrieved does not matter.

package.files

By default, the entire tree of files under the root of a package will be retrieved from the repository during “pakman get”. If a files property is specified, however, it identifies a subset of the tree that must be copied onto the local file system before building. This can be useful to reduce the amount of time and local disk space consumed by retrieving a directory when it contains a mix of needed and unneeded files.

The files property is an array of Perforce-style patterns. All of these patterns are relative to the package root, and they must refer only to files under the root (i.e., a leading “/” or “..” is not allowed).

files = {
   "src/...",
   "inc/...",
   "Makefile"
}

The files property does not establish multiple mappings into the client workspace. There is one such mapping per package, and that corresponds to the root of the package. When the files are retrieved, the directory structure beneath that root is consistent with that of the repository. The files property restricts only the scope of the “p4 sync” operations performed during “pakman get”.

The files property is most useful when dealing with packages that were not constructed with component-based builds in mind. When a directory that contains large numbers of files that are not needed by some consumers, a cleaner alternative to using the files property would be to factor it into distinct packages that live in separate directories. As separate packages, they could be independently substituted.

package.glue

Perhaps the most important function of Pakman is generating glue files. These are files generated during package retrieval and written into the package's directory tree. Typically they contain information about retrieved dependencies, allowing the package sources to remain free of any assumptions about where dependencies lie in the local environment, and allowing the build step to function without involvement of Pakman or the VCS server. At present, GNU Make is the only build tool for which support is built into Pakman, but the glue file concept is more general and Pakman could easily be extended to generate glue files for other languages.

Glue files are listed in the glue package property, which is a Lua array (a table with numeric indices starting at 1). There are two forms of entries that may appear in the glue array.

1. A string that gives the relative path (from the package root) to the glue file.

2. A table containing two fields: type and path.

When the first form (only a string) is used, Pakman replaces the string with the table form, setting path to the supplied string.

If no type is given, it will be set to the extension part of the path (or the entire file name if no “.” characters appear in path). If the glue file name matches “[Mm]akefile”, type will be set to mak.

Instead of a table, glue may be assigned to a function that will return a table. This function is passed the same environment parameter described in String Expansion.

For example, the following four lines are equivalent ways to define a single dependency:

1)  glue = { "x.y.z.min" }
2)  glue = { { path = "x.y.z.min" } }
3)  glue = { { path = "x.y.z.min", type="min" } }

There are two supported glue file types: min and mak.

min Files

A min glue file is an include file in makefile syntax. It defines variables, one per dependency, that hold the relative file system path to the result of that package. A project's makefile will typically include a min glue file and use those variables to refer to its dependencies. Consider this example pakfile:

deps = { A_DIR = "p4://server/depot/packages/a" }
glue = { "x.min" }

This will generate a Make include file that sets the variable A_DIR to a relative path to “//depot/packages/a”.

Templates

Packages may customize the format of the generated min glue file by assigning the template attribute of the glue entry. This attribute, when set, is a string that is expanded as described in the String Expansion section. In addition to the variables available in the standard string expansion context, the following variables are available when MIN file templates are evaluated:

• defs : a string that contains the lines of makefile syntax that assign dependency variables to relative paths

• vars : an array of names of the dependency variables.

• glueFile : an object from the glue array. See Package Objects, below, for a description of glue file objects.

Packages can append text to the default template by providing a template string that begins with the character “+”. The “+” will be replaced with the default template.

Here is the default template for MIN glue files:

# pakman min file

#{defs}

# adjust paths to be relative to current working dir
_pkg_deps = #{vars}
_pkg_here := $(filter-out ./,$(dir $(lastword $(MAKEFILE_LIST))))
$(foreach v,$(_pkg_deps),$(eval $v := $(_pkg_here)$$($v)))

# assign these variables only for the top-level makefile
ifeq ($(origin __pkg_dir),undefined)
  __pkg_dir     := $(_pkg_here)#{toroot}
  __pkg_result  := $(__pkg_dir)$(filter-out /.,/#{pkg.expanded.result})
  __pkg_deps    := $(_pkg_deps)
endif
__pkg_uri     ?= #{pkg.uri}
__pkg_version ?= #{pkg.version}

First, variables are assigned file paths that point to dependencies relative to the directory that contains the glue file.

Next, the paths are adjusted to be relative to the current working directory, allowing this MIN to be included from makefiles in other directories. The $(foreach ...) expression re-assigns each dependency variable, prepending the path to the directory containing the glue file.

Finally, a set of variables are assigned that convey properties of the package. These are defined conditionally so that they will not be reassigned by other glue files from other projects, which might occur if a makefile includes makefiles exported by its dependencies (see “transparent packages” in Build-time and Run-time Dependencies). The conditional assignment guarantees that these variables describe the first (or top level) package in that invocation of make, so each project's makefile can use them safely.

For very simple packages whose makefiles and glue files live in the same directory, the following minimal template might suffice:

glue = { { path="x.min", template="#{defs}" } }

MAK files

A glue MAK file is a makefile generated within a package that knows how to build the package and all its dependencies in the proper order. Consider this example pakfile:

deps = { A_DIR = "p4://server/depot/packages/a/pak" }
mak "x.mak"
commands.make = "make"

After retrieving this package, a user can build the entire tree (the package and all of its dependencies) by typing:

$ make -f x.mak

A MAK file defines two targets, tree and tree_clean, that build (or clean, respectively) the entire tree of packages under the current package. The tree target always comes first, making it the default target. The name tree is used instead of all to avoid conflicting with the target name that would typically be used in the package's own makefile. This allows a developer to include the MAK file from the package makefile, so that “make tree” (with no “-f ...”) could be used to build the entire tree.

A MAK file also defines a target for each dependency, so a user can easily rebuild or clean a specific dependency. Dependency variable names are used as the target names. With the above example pakfile, a user could rebuild the A_DIR dependency by typing:

$ make -f x.mak A_DIR_clean A_DIR

By default, a MAK file will conceal the output of the build steps for each sub-package, unless it encounters a build error. Specifying “VERBOSE=1” on the make command line will show all output from sub-makes. To generate a makefile that is always VERBOSE, add a verbose property with a value of true to the glue entry.

glue = { { path="x.mak", verbose=true } }

MAK files assume that they are invoked from the root directory of the project.

Glue File Conflicts

When multiple pakfiles refer to the same root directory, or when Build Variants are implemented using package parameters, two different package objects might attempt to write to the same glue file. This does not present a problem when both package objects specify the same contents for the file. If they specify conflicting contents Pakman will display a warning message, which should be taken as a sign of a serious problem: it indicates that both packages cannot coexist and still produce reliable built results.

package.mapping

Packages can use the mapping property to influence where in the local file system they and their dependencies will land when Pakman creates local mappings. This variable may be assigned a value of type function. For example:

mapping = pmlib.mapShort

For compatibility with older versions of Pakman (pre-0.90) you can wrap this inside an 'if' statement. Older versions will silently ignore the following instead of exiting with an error message:

if pmlib then mapping = pmlib.mapShort end

Pakman makes two built-in mapping functions available to package files:

• pmlib.mapLong maps each package under a directory called “/pakman”, including all of the path elements in the URI path. For example, if the workspace root is c:/p4 and the repository location is //depot/project/main, then the package will be placed at c:/p4/pakman/depot/project/main.

• pmlib.mapShort maps to names of the form “/pkg/<dir>”. It attempts to construct short yet meaningful directory names using just one path element from the URI. It also provides a list of longer alternative names that incorporate more path elements separated by “-”. These alternatives will be tried when the most preferred name is unavailable. Certain directory names (“main”, “latest”, “dev”, “rel”, “tip”, and “head”) are treated as less valuable, and directory names beginning with a digit are considered valuable but insufficient on their own. For example, if the workspace root is c:/p4 and the URI is p4://server/a/b/c/main/latest, then the package will be placed at c:/p4/pkg/c, unless that directory name is already used, in which case it will try c:/p4/pkg/b-c, then c:/p4/b-c-main, and so on.

  The pmlib.mapShort function can be used to avoid overly long file paths that result in command lines that exceed the limits of the underlying OS. Users can manually work around those problems by constructing a workspace view that has short local paths, but pmlib.mapShort provides a generic solution that works in automated builds that rely on Pakman to construct the workspace view.

When a package has no mapping property, its mapping defaults to the mapping rule that was used for its parent. When a package has two parents that use different mapping rules, its default mapping is undefined.

If the topmost package in the tree does not specify a mapping rule, its mapping function will default to pmlib.mapLong unless --mapshort or config.mapping is used to override it. The command-line option takes precedence over the configuration file.

If all of the alternative locations chosen by the mapping function collide with existing mappings, Pakman will try to manufacture a unique mapping by appending suffixes “-2” or “-3” and so on, until an available client location is found.

Even when a mapping property is specified, packages should never make assumptions about where their dependencies lie in the local file system. They should still use makefile variables to maintain Location Independence. Keep in mind the following:

• The mapping preferences apply only when Pakman creates a mapping. This behavior is described in Workspace Management.

• If two packages have the same most-preferred location, only one can win. Pakman will pick less-favored alternatives or manufacture a unique name in order to successfully map packages.

Custom Mapping Functions

In addition to the built-in mapping functions, package can provide their own implementations that match the following calling convention:

map(pak) -> names

• pak = the package being mapped (see Package Objects).

• names = an array of local paths (strings) or nil.

  The array is ordered from most-preferred to least-preferred. Each path begins with “/” and describes the local file system location relative to the workspace root for the package's VCS server. Pakman will use the most preferred name that does not conflict with an existing map.

  If names is nil, it indicates that the default mapping rule should be applied.

For example, when the built-in function pmlib.mapShort maps the package “p4://server/a/b/dev/1.0@999”, it will find pak.rootPath to be equal to “/a/b/dev/1.0”, and it will return { "/pkg/b-1.0", "/pkg/b-dev-1.0", "/pkg/a-b-dev-1.0" }.

package.message

When a pakman get command completes, it displays a message describing where the package is and the values of commands.make or commands.maketree.

Pakfiles can override the default message by setting the message field to a string or a function that returns a string. If a function is provided, it will be called after all packages have been retrieved, with the requested package (the one typed on the command line) as a parameter.

package.params

If the URI for the package being processed includes parameters, params will hold a table describing the parameter names and values, as parsed by pmlib.uriParse. If no parameters are present, params will be an empty table.

For example, when the package “p4://server/pkg/pak?debug;opt=2” is processed, the pakfile “/pkg/pak” will be executed with params[1] equal to "debug" and params.opt equal to "2".

Pakfiles may modify params to construct a “normalized” or “canonicalized” set of parameter values. The resulting params value will be used to re-construct the package object's uri value after pakfile execution completes.

As a convenience for pakfiles, params will accept a schema and perform validation of the parameters. To validate parameters with a schema, call params as a function, passing it the schema as an argument. It returns all of the positional parameters as separate return values. The following example illustrates a package that has two positional parameters:

local target, flavor = params {
   { "win", "mac" },
   { "debug", "release" }
}

More formally, a schema for parameters is a table that maps each supported parameter name or index to a parameter type. Each parameter type is a table with the following fields:

• values : a list of allowed values for this parameter (array of strings). If this array is empty, all values (except nil) are accepted. If values is nil, the array elements of the type table itself will be used instead (as in the above example).

• default : the value to set this parameter to if it is nil.

• optional : if true, nil is an allowed value for this parameter.

• alias : another parameter name that can be used to set this parameter. If a parameter is nil and its alias is non-nil, the alias value is substituted for the parameter and the alias parameter is removed from params.

Validation will throw an error if it encounters any parameter that is not named in the schema. Here are some examples:

params {
   a = { values = {"x", "y"} },       -- params.a must be either "x" or "y"

   b = { "x", "y", default = "x" },   -- params.b must be "x" or "y" or nil.
                                      -- If nil, it will be set to "x".

   c = { },                           -- params.c may hold any value, but it
                                         must be present in the URI.

   d = { default="x" }                -- params.d may hold any value.  If nil,
                                      -- it will be set to "x".

   e = { optional = true }            -- params.e may hold any value, including nil.
}

Due to the way positional parameters are encoded in the URI, we cannot represent nil as a value distinct from "" (the empty string). For example, when "pak?;;b" is decoded, parameters 1 and 2 will hold empty strings. In order to allow defaults to be easily applied, positional parameters with the value "" or nil will take the default value.

Refer to Build Variants for further discussion on the use of package.params.

package.redir

The redir property is used to redirect Pakman to a different package, which will be used instead of the current package.

redir = "p4://server/depot/dir/pak"

If a pakfile assigns the redir property, then after it finishes executing Pakman will destroy the just-created package object and substitute the new package object in its place.

package.result

Each pakfile can specify a result path. This is the location that will be visible to other packages that depend on that package. A package description may specify the result by assigning the variable result, as in:

result = "built"

For example, if package A includes the line:

deps = { ToolsDir = "/src/tools/srcpak" }

and “/src/tools/srcpak” specifies result = "built", then the min file generated for package A will assign ToolsDir the location of /src/tools/built, not /src/tools.

A result path allows a package to hide its internal structure. A package can place all of its outputs under a single directory (or in a single file) and expose only that directory to its clients. Consumers of the package only care about the result, and they do not need to know where the result directory lives within the package's tree. Among other benefits, this allows us to interchange source and pre-built (binary) versions of a package. A pre-built version can be represented as a plain package with no build step, and no dependencies.

In order to ensure interchangeability and facilitate build automation, the result should specify a directory, not an individual file, and all of the output of the package should live underneath the result directory. Consumers of packages should not make any assumptions about where the result directory lives, or what files reside outside of and relative to result directory.

By default, the result path is “.” (the same as the root directory for the package).

package.root

Each package has a “root”, which is the repository location under which all package sources lie. The root is the directory that must be mapped into the local client when Pakman retrieves the package (see Workspace Management).

The root is also used as the base directory for resolving other relative paths in the package, such as those for dependencies.

By default, the root of a described package is the directory that contains the pakfile. If you would like to store a package file somewhere other than the root directory of the package — in a sub-directory, or perhaps completely outside of the package tree — then you will have to specify the root explicitly in the package file.

Pakfiles specify the location of the package they describe by assigning the variable root to a repository location:

root = "p4://host/path"

Relative or absolute URIs may be used. If a relative location is used for the root, it will be resolved relative to the URI of the directory that contains the package file. If not specified in the package file, the root property defaults to “.” (the same as the directory containing the package file).

package.shared

This is a table that can be used by multiple instance of a single pakfile to share data. This is useful when the pakfile uses package parameters to specify Build Variants.

String Expansion

Various strings, such as glue templates, are “expanded” after the package file is processed. Instances of “#{<expr>}” are replaced with the value of <expr>. The string “#{#}” is replaced with “#”. The string <expr> may only contain one or more names separated by “.” characters. Arbitrary Lua expressions are not supported. When an array of strings is expanded into a string, the values are concatenated with a space character separating them.

When these expressions are evaluated, the following variables may be referenced:

• pkg : the package object (as defined by the pakfile) for which this file is being generated. See Package Objects below.

• paths : a table mapping dependency variable names to relative paths to their result directories.

• toroot : relative path to the root directory.

When glue file templates are evaluated, paths (in paths or toroot) are relative to the directory containing the glue file. In other contexts, the paths are relative to the root directory of the package.

If a function is provided instead of a string, the function will be called with a parameter which is a table that holds the above values. For example:

result = function (env) return env.paths.B end

... is equivalent to ...

result = "#{paths.B}"

Deprecated Features

You might encounter pakfiles that use an older syntax. For backwards compatibility with early versions of Pakman, the following functions are still supported, but creators of new package files are advised to avoid them.

• get(<table>) : merge <table> into the deps table.

• cmd(<make>,<clean>) : assign commands.make and commands.clean.

• min(<file>) : add <file> to the end of the glue array.

• mak(<file>) : add <file> to the end of the glue array.

• glue:Append(<item>) : add <item> to the end of the glue array.

Pakfile Environment

When a pakfile is executed, its environment is set to its own package object, so global variables assigned by the pakfile — such as deps or glue — end up in its package object. The package object's __index “metamethod” is set to point to the “globals” table, so Lua library functions like string.match and table.insert are available as usual in Lua. Pakfiles can use the variable _G to obtain a reference to the globals table, and self to obtain a reference to the package object itself. Pakman also provides two global tables, pmlib and sys, and a few other globals, described below.

pmlib

The pmlib table provides utility functions. It contains the following members:

• hash( str, [len, [alphabet])

  This function can be used to generate a short string from potentially long strings. len defaults to 5, and alphabet defaults to a 35-character string consisting of all numerals and lower-case letters (except for lower-case L). Example usage:

  result = pmlib.hash(uri)
  

• mapShort and mapLong: see the package.mapping section for a description of these functions.

• uriParse and uriGen: These functions convert between string and table representations of Pakman package locations. uriParse returns the table form; uriGen returns a string in normalized URI syntax. Either function will accept tables or strings as inputs.

  If two arguments are given to either function, the second one represents a base URI to be used to resolve the first URI.

  The table form of a URI contains the members scheme, host, path, version, params, and fragment. These fields may be nil. params, when present, is a table mapping parameter names or positions to values. Aside from scheme and host, any strings that appear within the table representation may contain arbitrary strings. They will be percent-encoded/decoded when a URI string is generated/parsed. Scheme names may only contain the alphanumeric characters and “+”, “-” and “.” (as per the URI specification). Host names are left in their percent-encoded form so that they may be interpreted in a scheme-specific manner.

  These functions follow the relative URI processing and normalization rules described in RFC 3986 with the following exceptions:

  • When combining relative URIs — in which the path does not begin with a "/" character — uriGen and uriParse preserve the property of associativity. For example, uriGen(uriGen(r1, r2), base) produces the same result as uriGen(r1, uriGen(r2, base)). This allows you to combine two relative URIs to generate a new relative URI.

  • Strings returned by uriGen will have parameters ordered deterministically, and no two parameters may have the same name. Percent-encoding is made uniform. This preserves everything semantically significant to Pakman and nothing else, so equivalent URIs will generate identical strings.

  • Pakman's version field is supported. It will be inherited from the base URI if not present in the resolved URI.

  • Percent encoding of all characters is normalized. This erases the distinction between encoded and un-encoded characters (e.g. “/” and “%2F” in the path). The intent is that these functions are to be used only with schemes that represent file paths.

  Examples:

  pmlib.uriParse("p?a;x=%3A%2F%3F")                 -->  {path="p", params={"a", x=":/?"}}
  pmlib.uriGen{ params={"a", x=":/?"} }             -->  "?a;x=%3A%2F%3F"
  pmlib.uriGen("?x=:/?&a")                          -->  "?a;x=%3A%2F%3F"
  pmlib.uriGen("a", "p4://host/base")               -->  "p4://host/a"
  pmlib.uriGen("a", "p4://host/base/")              -->  "p4://host/base/a"
  pmlib.uriGen("a", "../")                          -->  "../a"
  pmlib.uriGen("a?x", {path="base/", version="1"})  -->  "base/a@1?x"
  pmlib.uriGen("?x=1", "/a/pak")                    -->  "/a/pak?x=1"
  

sys

The sys table describes the version of Pakman. It contains the following members:

Global Functions

• readfile accepts one argument, a relative or absolute URI, and returns the contents of the file. On error, it returns nil and an error message. This reads the file directly from the VCS server and does not sync it locally. Relative URIs are treated as relative to the pakfile.

• require, loadfile, and dofile are Pakman-provided alternatives to the standard Lua functions. These work like the standard Lua functions except that they operate on URIs. This require implementation does not search the module path described in the LUA_PATH environment variable. When relative URIs are passed to these functions they are interpreted as relative to the URI of the source file. (Technically speaking, each source file gets its own instance of each of these functions.)

  Source files that are loaded and executed via these functions will see the same set of globals as pakfiles (pmlib, sys, etc.).

Package Objects

Package objects are constructed by pakfiles and may be examined by Hooks or Templates.

After the pakfile is executed, Pakman normalizes and resolves relative URIs. In addition to the properties initialized by pakfiles, the following properties are available when glue file templates are evaluated:

For “p4:” URIs, the version property indicates what version was specified when retrieving files from the server. This will be the version field from the URI when one is specified. Otherwise, it will be the most recent changelist in the depot as indicated by “p4 changes -m 1 //...”.

After processing the pakfile, Pakman ensures that each glue file entry (package.glue[n]) is an object with the following properties:

Both fsPath and fsDir are absolute paths in the file system namespace (not URIs). The data property is visible only when the packageGlue hook is executed.

Configuration Files

On start-up, Pakman looks for a configuration file named “.pakman” in the current directory (or if not there, in any of its parent directories). You can use the --config=<name> option to set the name to something other than “.pakman” or to disable this feature. Configuration files are Lua files that may assign one or more variables that control operation of Pakman, or register Hooks.

Configuration files are local to the build machine and are intended to contain information specific to the local development environment, which might differ from user to user or from time to time. Packages should not rely upon or dictate special configuration file settings.

Configuration Variables

config.verbose

verbose = true

config.mapping

mapping = pmlib.mapShort

config.vcs.p4.command

vcs.p4.command = "p4 -c pmclient"

vcs.p4.command = {
   acme123 = "p4",
   ["acme123/deploy"] = "p4 -c deploy",
   aswp401 = "p4 -p aswp401:1666",
}

vcs.p4.command = {  [""] = "p4 -c myclient" }

vcs.p4.command = "p4 -c myclient"

Hooks

Configuration files may define hooks or callbacks to be called while Pakman performs its work.

packageGlue hook

local function writeVars(pkg)
   local f = assert(io.open(pkg.fsRoot .. "/.cdvars", "w"))
   for var,p in pairs(pkg.children) do
      f:write( var .. "|" .. p.fsRoot .. "\n")
   end
   f:close()
end
addHook("packageGlue", writeVars)

varcd() {
  if [ -n "$1" -a ! -d "$1" -a -r .cdvars ] ; then
    IFS='|' && while read a b ; do
      if [ "$a" == "$1" ] ; then break ; fi
    done < .cdvars
    'cd' "${b:-$1}"
  else
    'cd' "$@"
  fi
}
alias cd=varcd

onVisit hook

Use Cases and Strategies

Build Variants

There are two ways to describe how different build results (variants) are produced from a single source tree. One way is by writing multiple pakfiles that reside in the same directory or specify the same root directory. The other way is by writing a pakfile that recognizes package parameters.

It helps to keep in mind that in Pakman the following things are independent and do not necessarily relate one-to-one with each other:

• source tree: a directory tree of files to be retrieved from VCS.

• pakfile: a file containing Lua code that constructs packages.

• package: an object with the properties described in Package Objects, including those that specify the make command, result directory, and source tree.

Build Variants with Parameters

Pakfile parameters can be specified in a URI as described in Package Locations. When a pakfile executes, it can access the parameters from its own URI by examining the package.params table. When a package file is included multiple times with different parameters, its pakfile will be executed multiple times, once for each distinct set of parameters. Each time the pakfile executes it constructs a new package object.

Packages are thereby free to define their own “contracts”: which options are supported, and what they signify.

Here is an example pakfile, named clib/pak, that describes two variants:

local v = "Win32_Release"
if params.debug then
   v = "Win32_Debug"
   deps.DbgMalloc = { path="../dmgmalloc", params={debug="1"} }
end
commands.make = "make V=" .. v
commands.clean = "make clean V=" .. v
glue = { v .. ".min" }
result = v

Here is a consumer of clib:

deps = {
   CLibDebug = "../clib/pak?debug=1,
   CLibRelease = "../clib/pak",
}
glue = { "x.min", "x.mak" }

The two dependencies identify different variants because the parameters they specify (or omit) are recognized by clib/pak and treated differently. In the min glue file, the variables CLibRelease and CLibDebug will point to different directories: <clib.root>/Win32_Release and <clib.root>/Win32_Debug.

Unrecognized Parameters

What if the parent package were to add the following dependency?

CLibOptimized = "../clib/pak?opt=1"

This specifies an option that is not recognized by the pakfile. Looking at clib/pak you can see that it will produce the same results for CLibOptimized and CLibRelease. As a result, these URIs effectively work as two aliases for the same dependency. The corresponding min file variables will point to the same directory, and there will be one make command in the mak file.

The build should work just fine, but Pakman will warn of a glue file conflict because the __pkg_uri variable in x.min gets assigned a different value in each 'variant'. We can resolve this problem by normalizing package.params:

params.opt = nil

This will cause the 'opt' parameter to be silently ignored, and removed from the package's uri field. This would be appropriate when the clib developer is aware of the optimization parameter and has deemed that the build results will be usable by clients that specify “opt”.

Additionally, the clib package may choose to provide early errors when clients use unrecognized parameters:

params {
   debug = { "", "1", default = "" }
}

Guidelines for Variants

When you create pakfiles that describe variants, you should understand and keep in mind the following concerns:

• Avoid glue file collision.

  Glue files are created by Pakman in the source tree, under the package root directory. If two variants name the same glue file but specify different dependencies, etc., which affect the contents of the glue file, a conflict arises. Pakman cannot accurately construct a package tree that includes both of those variants. It will detect such conditions and print a warning.

• Avoid result collision.

  Each variant should have its own result directory.

  If two different build variants are destined for the same result directory and they overwrite each other's files, then the build could produce unpredictably corrupted results.

  If two build variants share the same result directory but are carefully constructed not to overwrite each other's files, it presents other, less immediate problems. For one, downstream packages could easily become dependent upon build results they did not explicitly request, leading to build problems when packages are combined in different ways.

  Pakman does not currently warn of these conflicts.

• Avoid build collisions.

  The build command for each of the variants will be issued in the same root directory and operate within the same source tree. Each build should not be influenced by build operations performed for other variants. For example, temporary files generated when building one variant should not be able to affect the results of another variant built later. Good, common sense practices for constructing makefiles should be employed, including ensuring that all output and temporary files are written to a separate directory, specific to the variant, and not the source file directories.

  Pakman cannot detect these kinds of conflicts, since Pakman does not know about the details of the build steps.

Multi-Server Projects

The config.vcs.p4.command variable allows a user to associate different Perforce command strings with different host names by assigning a variable in a .pakman file. These command strings will need to specify server-specific configuration options. Below is a summary of some relevant p4 command-line options. Type “p4 usage” or refer to Perforce documentation for more information.

• -p <port> : This provides the host name and port (typically 1666).

• -c <client> : Client name.

• -u <username> : User name.

Pakman uses the domain name reported by the server in p4 info to validate the command table mappings.

Directory Organization for Multiple Servers

You must take care to ensure that the client workspaces used with the different servers do not have root directories that overlap in the local file system. The client roots should be different from each other, and one should never be underneath another one. If two clients were to overlap, the syncs performed in one client could overwrite and corrupt the files in another client.

A simple, reliable arrangement is to choose a directory for your multi-server “workspace”, and place client root directories for each server immediately under that directory, as shown below:

The top level directory is a good place for the .pakman file. This will cause Pakman to find the “right” settings for that tree whenever you are working in that tree, since Pakman searches upwards from the current working directory for the .pakman file.

Cross-Server Versioning

Since changelist versions are meaningful only within a single server, time-based versioning should be used when specifying package versions in a package tree that spans multiple servers.

Using Proxies with Multiple Servers

When one or more of the multiple servers is accessed via a proxy server, the proxy server address should not appear as a key in the command table. The keys in the command table match host names in URIs, which refer to the host server, not the proxy server. The proxy server address will appear, however, in the “-p” option to be passed to p4. For example, when accessing a server p4d via a proxy p4proxy, URIs should begin with p4://p4d/ and the server configuration may be specified in the following way:

vcs.p4.command = {  p4d = "p4 -p p4proxy:1666" }

Multi-Client Projects

When you are working on multiple independent changes to a project in Perforce, you may need to have two copies of the project directory on your local disk, which will require two client workspaces. By default, Pakman operates entirely within one workspace, so when you get a project tree all the dependencies will be mapped and retrieved into that workspace. Since some of these dependencies may be very large and relatively inactive — for example, a large tool chain — having a copy per workspace may be unnecessarily wasteful.

The config.vcs.p4.command variable allows different parts of the repository to be associated with different client workspaces, so a single get operation can construct a project tree that spans workspaces. This allows large common dependencies to be placed in a workspace that will be shared. If the repository tree under “/deploy” includes a number of mostly stable sources and binaries, multiple development workspaces can make use of a “.pakman” file with the following contents:

vcs.p4.command = {
   ["/deploy"] = "p4 -c my_deploy_client"
   [""] = "p4"  -- default
}

In this example, the first entry in the command table has an empty host field and a path consisting of /deploy. The empty host name will match any host field in a URI, which is valid only when all URIs reside on the server that the default p4 command line environment uses. Any packages that reside under /deploy will be retrieved using the “my_deploy_client” client, and other packages will be retrieved using the default client.

By maintaining different Perforce configurations for each project tree — for example, by using the P4CONFIG environment variable — a .pakman file with these contents will allow multiple project trees to share the same copy of the /deploy/... tree.

Package Guidelines

Location Independence

A package should not make assumptions about where it lands in the client workspace, or about where other packages or external dependencies lie in the local workspace. Instead, all external dependencies should be contained in packages, and their locations should be identified using variables in glue MIN files.

Packages that make assumptions about the client workspace will immediately exhibit one downside: in order to successfully build the package, each user must ensure that those assumptions are maintained. Even worse, such packages are not suitable for inclusion in other higher-level packages.

Some specific scenarios that we might want to enable include:

1. Inclusion in a larger tree. Any package tree should be able to be included in another package so that the overall build product can be automatically built. The higher-level package may include multiple package trees, and these may need to share common dependencies. When multiple package trees make assumptions about local directory tree structure, those assumptions may conflict with each other.

2. Substitution of dependencies. Many situations call for replacing a dependency somewhere in the tree with an alternative implementation or an alternative build step. Packages should therefore not hard-code the location of any dependency. Cases include:

   1. Building with a development sandbox version of a sub-package.

   2. Switching between different versions of a package in order to isolate the root cause of a problem.

   3. Switching between pre-built and built-from-source forms of a package.

3. Coexistence with external requirements for workspace views. Assumptions about the target system — e.g. specific drive letter mappings — can conflict with other requirements place on client systems.

Any explicit use of “..” within a makefile to escape out of the package's root directory is evidence of a problem in the package. Likewise, any use of absolute paths to refer to dependencies is evidence of a problem. When parts of a package refer to files outside of the package without using a variable defined in a glue file, an assumption is being made about the local development workspace.

In practice, location independence is not difficult to achieve, because glue files make available the locations of dependencies as make variables. Even in a “Big Tree” development environment, it is basic common sense to reference external dependencies with variables, since file organization tends to change over time, and variables can minimize the impact this has to makefiles and other build scripts.

Minimize Environment Dependencies

Packages should minimize their dependencies on the local environment: build system setup, installation requirements, or OS dependencies. Since these types of issues confront just about every package, the logic that deals with these issues (e.g. auto-detection or validation of the environment) should be separated into a package of its own.

A package's build step should not depend on user-supplied environment variables, unless that package's role is detecting and validating the environment. Environment variable assumptions inject manual steps into the processing of setting up a build environment, they can introduce otherwise avoidable conflicts between packages, and they pollute otherwise portable packages with details that can be factored out into a platform-specific package where they [Use of environment variables within a package's build step, as a means to communicate between parent and child processes, however, presents no such problem.]

Minimizing Coupling

All else being equal, a package with fewer dependencies is a better package, because it is more easily reused in other contexts, and will be simpler to comprehend, analyze, modify, and audit. That being said, placing complexity in a separate package and listing it as a dependency can improve flexibility. Minimizing overall complexity is the key goal here.

Similarly, packages should minimize their surface area — the sum of the ways in which the package interacts with its consumers. Surface area can be defined as the complexity of the package's interface description: the contract its build product must meet.

Switching Dependencies

Building on the example discussed in the Introduction we can modify the project to swap one package out for another. Edit the pakfile for runlua (from the pakman directory you can type “make '$(RunLuaDir)'” or examine the x.min file find out where this is in your file system). Comment out the line that references builds/pkgs/lua-5.1.4.pak and un-comment the line that references lua/lua-5.1.4/pak, and save the result to your local disk.

Those two lines should now look like this:

-- LuaDir = "../../builds/pkgs/lua-5.1.4.pak",          -- Lua 5.1.4 pre-built
LuaDir = "../../opensource/lua/lua-5.1.4/build/pak", -- Lua 5.1.4 from source

Reissue the same pakman get command we used earlier to update the project tree. Since we have already retrieved Pakman locally, it will be easier to use a relative path than the long absolute URI. Assuming your current working directory is still the “pakman” directory, you can type:

$ pakman get @926500
Client bhk-sedition-pt saved.
Getting @926500
*** Using locally edited c:/p4/pt/pakman/depot/users/bhk/proto/runlua/pak
mapping //depot/users/bhk/opensource/lua/lua-5.1.4 --> //bhk-sedition-pt/pakman/depot/users/bhk/opensource/lua/lua-5.1.4
updating client bhk-sedition-pt
syncing p4://acme123.com:1666/build/tools/Vc7/...@926500
syncing p4://acme123.com:1666/build/tools/WinNT/...@926500
syncing p4://acme123.com:1666/make.d/...@926500
syncing p4://acme123.com:1666/depot/users/bhk/opensource/lua/lpeg-0.9/...@926500
syncing p4://acme123.com:1666/depot/users/bhk/opensource/lua/lua-5.1.4/...@926500
 [106 changes]
syncing p4://acme123.com:1666/depot/users/bhk/opensource/lua/luafilesystem-1.4.2/...@926500
syncing p4://acme123.com:1666/depot/users/bhk/proto/luau/...@926500
syncing p4://acme123.com:1666/depot/users/bhk/proto/make.d/...@926500
syncing p4://acme123.com:1666/depot/users/bhk/proto/pakman/...@926500
syncing p4://acme123.com:1666/depot/users/bhk/proto/runlua/...@926500
syncing p4://acme123.com:1666/depot/users/bhk/proto/simp4/...@926500
writing c:/p4/pt/pakman/depot/users/bhk/proto/pakman/x.min
writing c:/p4/pt/pakman/depot/users/bhk/proto/pakman/x.mak
writing c:/p4/pt/pakman/depot/users/bhk/proto/runlua/x.min
writing c:/p4/pt/pakman/depot/users/bhk/proto/runlua/x.mak
writing c:/p4/pt/pakman/depot/users/bhk/proto/make.d/x.min
writing c:/p4/pt/pakman/depot/users/bhk/opensource/lua/lua-5.1.4/build/x.min
writing c:/p4/pt/pakman/depot/users/bhk/proto/simp4/x.min
writing c:/p4/pt/pakman/depot/users/bhk/proto/simp4/x.mak
writing c:/p4/pt/pakman/depot/users/bhk/proto/luau/x.min
writing c:/p4/pt/pakman/depot/users/bhk/proto/luau/x.mak
Done.  10 packages retrieved.
To build:
    cd c:/p4/pt/pakman/depot/users/bhk/proto/pakman
    make -f x.mak            # builds the package and its dependencies
    make all                 # builds just the package

This time, Pakman retrieved one new package. The already-retrieved packages were left in place. The pre-built Lua package (which we swapped out) also remains in the workspace, but it is no longer referenced in the project tree. It will not have to be retrieved again if you swap it back in.

Now when we rebuild Pakman we will see an additional build step:

$ make
making ../../opensource/lua/lua-5.1.4
making ../runlua
making ../luau
making ../simp4
making .

At this point we can easily generate a new executable after making changes to the Lua sources or rebuild everything with different levels of optimizations or different debugging or profiling options.

Switching Dependencies at Build Time

While this works well for switching between pre-built deployments and source trees, you might want to be able to select between packages at build time, not when retrieving. For example, you may be on an airplane when you need to build from sources.

We can modify our example project to select between Lua-5.1.4 and LuaJIT-2.0 at build time. The build step is outside of Pakman's domain, so we are not talking about Pakman features at this point, but this use case illustrates how the same package construction principles can apply in other contexts.

First, we modify runlua/pak to include both Lua and LuaJIT at the same time. We need to use two different variable names for this. The relevant lines from the pakfile should look like this:

...
   --LuaDir = "../../builds/pkgs/lua-5.1.4.pak",      -- Lua 5.1.4 pre-built
   LuaDir = "../../opensource/lua/lua-5.1.4/pak",   -- Lua 5.1.4 from source
   LuaJITDir = "../extpaks/luajit/pak",
...

After modifying a pakfile we need to update the package tree using Pakman. This will retrieve the new dependency (LuaJIT) and update runlua/x.min to define both LuaDir and LuaJITDir.

$ pakman get @909090
...

Next, we add the following three lines to runlua/runlua.min, immediately after the line that includes x.min:

ifdef RUNLUA_USEJIT
   LuaDir = $(LuaJITDir)
endif

At build time we can rebuild Pakman either way using:

$ make clean all RUNLUA_USEJIT=1

or:

$ make clean all

When deciding whether to construct a pakfile that enables this type of behavior, consider that such a package has more dependencies, and will require a larger build tree than a more specialized package. This would not be appropriate for an automated product build, for example.

Finally, consider that you can have multiple pakfiles for a single package. One could build from binaries, one could build from source, and one could allow either to be selected. Users could retrieve the package appropriate for their task facing them.

Automating Pakman

Setting up an auto-build server for a Pakman-based project will involve invoking Pakman from a script. The script will have to perform the following tasks:

• Identify how to build the package.

  One approach to automating the build of a particular package would be to create a customized workspace view and write a script to execute a certain build command in a certain directory after retrieving the package with pakman get. This hard-codes some assumptions about the package's make command and where it lands in the workspace view.

  However, we can easily write a generic script that would be able to build any package given only its URI. This would accommodate any changes to the package's internal structure or its root directory location, and make no assumptions about the workspace view. We can do this by creating a temporary “wrapper” pakfile that declares the package(s) to be built as dependencies and specifies a glue MAK file. The automation script can then use the generated MAK file to build the desired package(s). This is like treating the automation script as a package in its own right, with its own dependencies, and its own rules for how to build its tree of dependencies. In bash or sh it would look like this:

  echo 'deps = { A="'$1'" } ; glue = { "x.mak" }' > tmp.pak
  pakman get tmp.pak
  make -f x.mak tree_clean tree > log.txt
  

  We can expand upon this by adding code to construct a temporary Perforce client for the duration of the build operation. You can refer to //depot/users/bhk/builds/scripts/pakmake for a bash script that automates clean builds “from scratch” in a way that can be triggered remotely via ssh.

• Determine whether any changes have been made to the project since the last build.

  A simple approach would be to hard-code a set of directories to be monitored using p4 changes -m 1.

  It is feasible to automatically and precisely determine the set of directories to monitor by traversing the tree of package files. Pakman may be extended in the future to report the most recent changelist, given a package URI.

  Using pakman show, we get a list of all packages that will be retrieved. A script can read that output and request the most recent changelist. It is up to the script to decide how to handle what a pakman get would request in terms of changelist and what is available in the P4 repository.

Archiving Builds

As the Switching Dependencies section illustrates, Pakman simplifies interchangeability of source and pre-built forms of packages.

When developing your component, you may ordinarily want to use the pre-built forms of its dependencies in order to minimize build times. Sometimes, however, you may need to modify or trace through the source of the underlying component — for example, when integrating with a new version of the dependency or debugging a failure. Also, you may want to build downstream components (clients of your component) along with the source form of your component in order to diagnose problems before releasing your package.

Build-time and Run-time Dependencies

In order to construct components that allow interchangeability of source and pre-built forms, we need to think about package dependencies and how they are used by the package.

In the simplest case, a package's dependencies are used only during the package's own build step. We will call these opaque packages. Opaque packages can be thought of as having only build time dependencies. After an opaque package is built, its dependencies are not referenced. Their results are a self-contained tree of files with no references to files outside of the tree. Opaque packages can easily be interchanged with a plain package (a tar'ed or zip'ed tree of previous build results).

In other cases, a package will reveal the locations of its dependencies to its own consumers, perhaps by exposing its own generated MIN file. Its consumers can then access its dependencies in-place when they build. We will call these transparent packages. Transparent packages can be thought of as having run time dependencies. Their dependencies need to be around even when downstream components are built. If you were to archive the result directory tree for a package as a plain package, it would not be interchangeable with the original package. The archived version would have to carry along dependency information as well.

Although transparent packages are not easily archivable, they are sometimes immensely useful for aggregating or extending other packages. A transparent package can act as a shorthand for a collection of packages. By re-exposing their dependencies in place they introduce no extra copies into the build process.

Another thing to consider is that when a package has no build step, there is not much value in archiving its build results. This leads us to the following observations and guidelines for archiving or warehousing:

• Purely opaque packages can be archived as plain packages (simple bundles of files without external dependencies).

• “Purely” transparent packages — that is, packages without their own build steps — do not need to have their results archived.

• Avoid using transparent packages that include their own lengthy build steps. It should not be difficult to re-factor such a package into purely transparent and purely opaque parts.

Bootstrapping Pakman

The prospect of bootstrapping Pakman — building Pakman on a new OS for which no Pakman executables currently exist — provides a good example of how to exploit the fact that a Pakman-constructed build tree is copy-deployable.

The problem is that Pakman is used to build Pakman itself. Without a functioning Pakman on the new OS, how can it be built?

The solution is to run Pakman on a supported platform (e.g. Windows) to construct a directory tree that includes the Pakman sources and all its dependencies. Then zip or tar up that directory tree, unzip it on the remote system, and build the package.

To construct such a directory tree:

1. Create a new Perforce client, edit it, delete its “View” section, and set its root to a new, empty directory.

2. pakman get p4:///depot/users/bhk/proto/pakman --p4="p4 -c <client>"
   

   If you are bootstrapping Pakman on a platform that has none of the pre-built binaries Pakman depends upon, you can substitute the build-from-source alternative packages before issuing this command.

3. Tar or zip the files under the client root.

./configure

Ideally, glue files describe everything a package needs to know about the outside world, either directly or indirectly. They can insulate packages from system, site, and local environment dependencies.

It is informative to compare this approach with the “autotools” (“./configure”) approach widely used in open source projects.

The ./configure approach relies on installation to make dependencies available to a package. Installation involves each package copying its outputs into a central location, so it is fraught with security compromises and poor software hygiene (possibility of naming conflicts, etc.). It also complicates switching between alternative dependencies, or working on two different projects that utilize different versions or branches of a package.

In a Pakman environment, we can express dependencies explicitly in a pakfile instead of relying ./configure to find them and relying on the user to have already installed them.

But what about the cases where even Pakman packages have to rely on installation and uncontrolled external dependencies? What about compilers or other system-supplied services, which for portability reasons or other pragmatic reasons, cannot be supplied as packages in VCS? Making assumptions is always dangerous. In these cases the best that can be done is to provide code that explores, evaluates, and validates the environment on the system ... just like ./configure.

In a Pakman environment, this detection and validation of the environment could be encapsulated in one or more packages. These packages would have build steps that perform the auto-detection and generate make include files or other files.

One upside of handling environment detection as a separate package is that only one copy is needed when multiple packages share it. Another upside would be that it would only have to execute once when it is shared by multiple packages. A third upside is that the environment detection package, once it is an external package, can be more flexibly managed. For example, one could easily could substitute an alternate implementation if the standard implementation does not work with the build system being used, or perhaps an optimized implementation with partially or completely pre-built results.

Alternatives to Pakman

“Big Tree”

One approach to managing a software project is to maintain a consistent, comprehensive project tree in VCS. Perforce supports this model well. This “big tree” approach can yield reliable and fully automated builds.

One downside with this approach arises when many independent code lines share ingredients (tools, compilers, system headers, etc.). Duplicating the shared resources, which can be quite large, into each big tree can get unwieldy.

Managing sandboxes and development branches when only a small sub-component is being modified becomes tedious and/or unwieldy. Branching a very large project tree to develop a localized change can be cumbersome, imposing a load on the server and on the local machine's disk space, and introducing into the development process VCS operations that are potentially very slow. The alternative of branching a sub-tree and constructing a workspace that overlays the sandboxes sub-tree within the main tree is tedious.

Limited or no Automation

This is the simplest and most common approach. In order to build the software, a user must know how to set up the client spec, install software on the build machine, and otherwise configure the local machine. These steps can theoretically be documented, but by their nature non-automated processes cannot be automatically tested and will tend to become stale and invalid. Even worse, the assumptions one project places on a build machine may conflict with the assumptions made by other projects.

Each build may also require manual steps — a specific set of 'sync' commands to be issued to the server — but these can be easily automated in a batch file or script (e.g. sync.bat).

Versus a “sync.bat” approach, Pakman offers the following benefits:

1. Platform independence.

2. Insulating a project from client spec mapping assumptions. Glue min files enable location-independent components, which avoid potential conflicts with other subsystems and enable easy substitutions of alternatives (sandboxes or alternate versions).

3. Interchangeability of packages. Pakman insulates a package from dependencies on the internal structure of packages that it consumes, such as where its results are produced, or whether a build step is required at all. This makes it easy to switch between a pre-built package or a package that is built from source.

4. Scalability. Recursive make execution times can grow exponentially with project size. Pakman-generated glue MAK files roll up the entire sub-tree into one makefile, introducing only one level of recursion.

5. Automation of client workspace construction. When a project is first retrieved, or as it evolves, each user does not have to manually add new lines to his or her client workspace.

6. Easing maintenance of projects. Package files eliminate double-entry of paths to dependencies, so makefile vars and sync.bat lines do not need to be kept in sync when developers make changes.

7. Validation. Pakman will warn when conflicting versions are requested, rather than silently corrupting the dependencies of some packages. Pakman provides reasonable error reporting when p4 invocations fail.

8. Hierarchical substitutions [Not Yet Implemented] : Alternative configurations can be even further simplified by allowing users and pakfiles to specify substitutions of packages that apply across an entire sub-tree of packages.

9. Construction of a copy-deployable build tree. The build tree constructed by pakman get consists of a set of directories that reference each other only via relative paths, and which have minimal dependencies on the environment. As a result, this directory tree can be copied — or zipped/tarred and then unzipped/untarred — elsewhere on the machine or on a different machine, where it can then be built.

10. Avoiding version skew. All 'sync' operations performed by Pakman specify a version. With unspecified versions, a 'submit' that occurs during the sequence of 'sync' operations could leave the user's (or build server's) workspace in a state inconsistent with any version on the server.

In addition to these benefits, there are a number of conveniences enabled by Pakman, and a number of smaller, use case-specific benefits that are made possible by having package inter-dependencies expressed in a machine-readable form. The examples given herein for user-supplied actions give just a hint of the possibilities.

Other Tools

There are a number of comprehensive, full-featured component-oriented SCM frameworks designed for supporting large-scale software development, including qpbuild and maven. Coming at the problem from a different direction there are package management tools used in Linux distros, such as APT, RPM, and pacman.

While it shares many of the same goals, Pakman offers a unique combination of features:

• Pakman avoids introducing dependencies on servers (aside from the VCS servers) or anything else that would impose administrative overhead. Pakman adoption requires no centralized administration.

• Pakman has no uncontrolled run-time dependencies or installation requirements. Pakman can be deployed as a single executable file ready to run on vanilla OS installations.

• Pakman limits its scope to the problem of retrieval and configuration of components. It does not involve itself in the mechanisms for building and testing the software. Source trees constructed by Pakman are buildable without Pakman, and can be deployed simply by copying.

• Unlike the Linux-oriented tools, Pakman avoids the notion of “installation” of components, so a developer can work with multiple products or multiple versions of a product on the same machine, switching between builds easily.

Troubleshooting

Frequently-Encountered Problems

1. Invalid P4 configuration.

   When using Perforce, Pakman relies on the p4 command-line client and depends on its settings, including your default Perforce server. Type p4 info to make sure that your command line Perforce client is properly configured. Refer to the Perforce documentation for guidance.

   If p4 info looks correct but Pakman operations still fail, use the --log=<file> or --verbose / -v options to see what Perforce commands are issued by Pakman. Then you can try those Perforce commands directly to see why they are not working.

2. “Cannot map package”

   In order to retrieve packages from Perforce, Pakman may have to add lines to the workspace view. Since Pakman never removes or shadows mappings in the view, pre-existing entries may present conflicts that the user must address. Since each mapping specifies two patterns, one for repository file names and one for local file names, there are two types of conflicts that may arise:

   1. A required repository location is already mapped into the workspace, but not completely. The resolution in this case is to manually edit the view to create a mapping that includes then entirety of the required packages.

   2. The local directory chosen by Pakman for a package is not available. That is, the directory or some files beneath it are specified in existing maps. This problem can be resolved by manually editing the workspace view to move files “out of the way” of the default client location, or by creating some other map for the required package.

   These problems can be avoided in the first place by starting with a completely empty workspace view. See Workspace Management for more information.

3. Cygwin P4 clients

   Perforce distributes two different versions of the p4 command line client for Windows: a “native” Windows version and a Cygwin version. Pakman will print a warning if it detects the Cygwin version, because it can cause problems for Pakman by reporting local file paths that are not valid Windows paths (e.g. “/cygdrive/c” instead of “c:/”). To avoid this warning and the potential problems, either remove the Cygwin version from your command path or point Pakman to a Windows native version of the p4 client by using the -p4 option of the p4.command configuration file setting.

Reporting Problems

If none of the above issues seem to identify the problem, or if the problem is not machine-specific (a problem with how pakman interprets package files), then:

1. Re-issue the failing command with the --log=<file> option. This will generate a log of the actions performed by Pakman, including its interaction with Perforce.

2. Raise an issue in the Github Issue Tracker, including the problem description and a log of the failing session. Please indicate whether this appears to be machine-specific problem.