These instructions are for users users of Onyx who want to work with and modify the source code of the project.
This file is the top-level readme for Onyx. It gives quick instructions about how to retrieve a copy of the project, and it points to other documentation within the project. These instructions should get you going on either the HLTCOE HPCC machines, on a Mac, or on Linux.
Note
some of the instructions below are out of date.
The project uses the Bazaar version control system. The Bazaar version control is often called bazaar or bzr and is also sometimes referred to as bazaar-ng (for next generation). See: http://bazaar-vcs.org/ You will need to install bazaar before you can retrieve the HLTCOE project.
The HPCC machines have Bazaar already installed in /usr/local/bin. Try “which bzr” and verify that you get “/usr/local/bin/bzr”. Then you can move on to “Installing the project” below.
On the Mac platform we recommend using the command-line program ‘port’ to retrieve and build OpenSource projects used by HLTCOE. To prepare your system for port, you need to install Apple’s Xcode SDK. Xcode is both an application and an SDK toolchain. It contains GCC tools which are needed by the port command when it builds packages.
Once you have Xcode installed, then install the MacPorts package. This installs tools under the /opt/local directory. It may also fiddle your path, e.g. in ~/.profile.
MacPorts has command-line tools that let you manage the installation of numerous OpenSource projects. The main tool is port:
$ port help
port
[-bcdfiknopqRstuvx] [-D portdir] [-F cmdfile] action [privopts] [actionflags]
[[portname|pseudo-portname|port-url] [@version] [+-variant]... [option=value]...]...
Supported commands
------------------
activate, archive, build, cat, cd, checksum, clean, compact, configure,
contents, deactivate, dependents, deps, destroot, dir, distcheck, distfiles,
dmg, dpkg, echo, ed, edit, exit, extract, fetch, file, gohome, help,
info, install, installed, lint, list, livecheck, load, location, mdmg,
mirror, mpkg, outdated, patch, pkg, platform, provides, quit, rpm, search,
selfupdate, srpm, submit, sync, test, trace, unarchive, uncompact, uninstall,
unload, upgrade, url, usage, variants, version, work
... [ for many lines ]
For more information
--------------------
See man pages: port(1), macports.conf(5), portfile(7), portgroup(7),
porthier(7), portstyle(7).
Use port to install Bazaar version control. Beware of a potential confusion: the unrelated GNU Arch project is called bazaar in the MacPorts databases; be sure to get bzr. In order to use port you must be root, e.g. via sudo. Using the following you will be prompted for the administrative password, then port will get to work:
$ sudo port install bzr
Note that port manages package dependencies and will have to download and build several other packages before it finishes. To see more about what it’s doing you can try verbose mode or even debug mode:
$ sudo port -v install bzr
$ sudo port -d install bzr
When port has finished its work, you should be able to run bzr in a (new) shell. E.g.:
$ bzr
Bazaar -- a free distributed version-control tool
http://bazaar-vcs.org/
Basic commands:
bzr init makes this directory a versioned branch
bzr branch make a copy of another branch
bzr add make files or directories versioned
bzr ignore ignore a file or pattern
bzr mv move or rename a versioned file
bzr status summarize changes in working copy
bzr diff show detailed diffs
bzr merge pull in changes from another branch
bzr commit save some or all changes
bzr log show history of changes
bzr check validate storage
bzr help init more help on e.g. init command
bzr help commands list all commands
bzr help topics list all help topics
The project is accessed from a repository on an HLTCOE server.
At present you need an account on the HLTCOE network in order to retrieve a copy of the project. You must also be a member the bzruser group. The IT folks can set you up with an account.
When accessing the server, it’s a huge convenience to use the ssh RSA authentication keys so that you don’t have to type your password each time you access the repository. See the ssh-keygen man page. On your local machine use ssh-keygen to generate the ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub files. The following examples show this process, where <user> should be replaced with your user name. Be sure to use an empty passphrase, otherwise you will be prompted for the passphrase each time you try to access the server.
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/<user>/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/<user>/.ssh/id_rsa.
Your public key has been saved in /home/<user>/.ssh/id_rsa.pub.
The key fingerprint is:
b6:a8:2a:a9:c7:5e:bb:a6:ad:08:03:73:e8:ed:2d:17 <user>@hltcoe.hodain.net
The key's randomart image is:
+--[ RSA 2048]----+
| |
| . |
|+ . E |
|ooo +. . |
|.. +.o S |
|. ..o. o . |
|+.. . . . |
|oooo.o .+ |
|oo+++ |
+-----------------+
Then, append the contents of the local ~/.ssh/id_rsa.pub to the server file ~/.ssh/authorized_keys:
$ ssh <user>@external.hltcoe.jhu.edu 'cat >> ~/.ssh/authorized_keys' < .ssh/id_rsa.pub
<user>@external.hltcoe.jhu.edu's password:
Then verify that you can access the account and run a command using ssh, e.g.:
$ ssh <user>@external.hltcoe.jhu.edu hostname
gpsrv5
Once you are set up to access the server, you can create a local branch of the project.
This will prompt you for a password (on the HPCC machines you may get prompted twice; this is OK) and then retrieve the project files, putting them into the directory my_onyx_code. Note that
You would use the following command with your user name in the <user> slot. This step may take a while to finish without giving any indication that anything is happening:
$ bzr branch bzr://onyxtools.bzr.sourceforge.net/bzrroot/onyxtools/onyx/ my_onyx_code
This will have installed a local branch of the project in the my_onyx_code directory.
The URL bzr://onyxtools.bzr.sourceforge.net/bzrroot/onyxtools/onyx/ refers to what is called the parent repository. It’s the version of the project (and all its history) from which the local branch is created.
The local branch consists of a working tree and a repository. The working tree is all of the files and data of the project, it’s what you see if you browse around in my_onyx_code. This is where you look at code and modify files, etc.
A repository is where bzr does all its recordkeeping. The local repository is found under my_onyx_code/.bzr.
Note
Do not change anything in the repository; it is the bzr database.
The project has several commands that use bzr tools to manage you local branch and the parent repository. These commands live in the op directory. They are typically executed from the top-level directory of the project. This convention makes it easy to work with several different installations of the project.
To get a summary of what’s new in the repository since you last synchronized with the repository. The snew command shows you the differences between the repository’s change file, which is changes.txt. If you are up to date you will get a short message to that effect:
$ op/snew
Local revision is 461, which is current for parent: bzr://onyxtools.bzr.sourceforge.net/bzrroot/onyxtools/onyx/
You are up to date!
If you are out of date you will get a more verbose message:
$ op/snew
Diffing ./changes.txt with parent version at bzr://onyxtools.bzr.sourceforge.net/bzrroot/onyxtools/onyx/
=== modified file 'changes.txt'
*** local changes.txt 2009-07-22 16:20:32 +0000
--- parent changes.txt 2009-08-11 18:46:20 +0000
***************
*** 15 ****
--- 16,112 ----
+ Commit at 2009-08-11 14:46:19 -0400 by <username> <<user>@<host>> from <branch-nick>
+ Message: Additions to top-level documentation
+ Last revno: 460 Last revision id: <revision-id>
+ Last revision date: 2009-08-11 13:26:33 -0400
+
+ modified:
+ doc/sphinx/SConscript
+ doc/sphinx/index.rst
+ doc/sphinx/toplevelorg.rst
+ added:
+ doc/sphinx/developer.rst
+ doc/sphinx/devo
+ doc/sphinx/user.rst
+ templates/restructured.rst
+ Updates to the documentation top-level structure.
There is also a less-informative lower-level command that shows you bzr’s view of what is different between your branch and its parent repository. You use missing to see what merge or pull would retrieve from the parent repository in order to bring your local branch up to date with the parent repository:
$ op/missing
Using saved parent location: bzr://onyxtools.bzr.sourceforge.net/bzrroot/onyxtools/onyx/
You are missing 1 revision(s):
------------------------------------------------------------
revno: 461
committer: <username> <<user>@<host>>
branch nick: <branch-nick>
timestamp: Tue 2009-08-11 14:46:20 -0400
message:
Additions to top-level documentation
When you are out of date, and you want to update, use the pull command to retrieve updated versions of files from the server:
$ op/pull
Using saved parent location: bzr://onyxtools.bzr.sourceforge.net/bzrroot/onyxtools/onyx/
+N doc/sphinx/developer.rst
+N doc/sphinx/devo
+N doc/sphinx/user.rst
+N templates/restructured.rst
M changes.txt
M doc/sphinx/SConscript
M doc/sphinx/index.rst
M doc/sphinx/toplevelorg.rst
All changes applied successfully.
Now on revision 461.
You can also use op/merge, which is like the pull command, but is used when there are change conflicts that need to be merged, and which may require your intervention to resolve.
To make changes to the project you just modify the files in question. In order to add a new file or directory to the project, use the bzr add command.
To see how bzr views the status of the project, use the status command:
$ op/status
added:
LICENSE.txt
NOTICE.txt
modified:
doc/presentations/logo-samples.ppt
py/onyx/__init__.py
unknown:
LICENSE.BSD.txt
usage-example.txt
doc/work/
This is a complex example:
Once you have a coherent set of changes that are tested and documented, and the build succeeds, then you are ready to check-in.
We use a small set of commands that we layer on top of Bazaar. The purpose of these commands is to automatically update the project file changes.txt which lives in the top-level of the project. Take a look at that file to see a prose history (most recent first) record of changes to the project.
If you have made changes to files in the project which you are ready to commit, here’s the procedure you should follow. Begin by doing:
$ op/prep
modified:
doc/presentations/logo-samples.ppt
py/onyx/__init__.py
added:
LICENSE.txt
NOTICE.txt
===== Additional bzr output - FYI only =====
unknown:
LICENSE.BSD.txt
usage-example.txt
doc/work
Renamed /Users/hugh/onyx/onyx_doc/comments.txt to /Users/hugh/onyx/onyx_doc/comments.bak
Wrote new template file: /Users/hugh/onyx/onyx_doc/comments.txt
Wrote new checkin file: /Users/hugh/onyx/onyx_doc/ckin_files.py
Now edit comments.txt then run ckin to complete your commit
You may also edit ckin_files.py if you wish to exclude some modified files from this check-in.
This will fill out the file comments.txt in the top level of your project with a template showing which files you’ve changed, added, removed, etc, based on the output of op/status.
Now edit comments.txt, adding text describing what you’ve done. Descriptions generally run from a one or two sentences to a few paragraphs, but occasionally longer entries are appropriate. See the project changes.txt for examples of the descriptive prose style you should add to comments.txt.
When you are done editing your descriptions of the work you’ve done to the project it’s a very good idea to run SCons one more time, just in case you’ve inadvertently changed something. If you have not changed anything this step doesn’t take long:
$ op/scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: building associated BuildDir targets: build
scons: `.' is up to date.
scons: done building targets.
Now you are ready to check in:
$ op/ckin '<log message>'
where <log message> is a short message for the bzr log. This command will automatically insert the contents of comments.txt at the top of changes.txt, and then it will commit the changes from your working tree to your local repository. This latter step is done via bzr‘s commit machinery.
At this point, your local branch should have no status:
$ op/status
When it’s time to propogate your commited changes to the parent repository, so that everyone else can retrieve them, do:
$ bzr push
This updates the server’s branch so that it is a mirror of your local branch. Once you’ve done this, when other developers of the project do op/snew, they will see you description of your changes, and they’ll do op/pull, or op/merge to propogate those changes into their local branch.
Top-level directories in the project often have a readme. Here are some of them:
The directory ./readme has some documentation that doesn’t seem to belong anywhere else, e.g. how to set up a Mac for the project.
In order to build and use the project, you will need some third-party tools. Here’s a list of what’s known to be needed.
Everything you need to build on the HPCC machines is already installed and should be on your path.
The Xcode toolchain is required. See: http://developer.apple.com/tools/xcode/
Python is retrieved, built, and installed via port. The list of available packages is huge (over 4000) so be selective when searching. E.g. there over 85 Python 2.5 related sub-packages:
$ port search py25 | wc
89
Here is a recommended set of Python tools:
$ sudo port -v install python25 py25-ipython py25-numpy
This will build and test numerous components and it takes a while to run (go get lunch).
We are trying to keep the number of environment variables you must set to a minimum. You should put the ‘py’ directory of the project on your PYTHONPATH. E.g. if you’ve installed the project in ~/my_onyx_code, then you would have the following in your ./profile or equivalent:
# Setting the path for python and MacPorts
export PYTHONPATH=~/my_onyx_code/py
You shouldn’t need to set any other environment variables on the HPCC.
You should verify that the port bin directories are on your PATH. You should have the following in your ./profile or equivalent:
# Setting the path for MacPorts
export PATH=/opt/local/bin:/opt/local/sbin:$PATH
Onyx is a build-based platform. That is, before using the tools you have to build and test them. The building is handled by the SCons software construction tool. A version of this tool is included in the project so you do not have to install it. The build and test process is unified. That is, when SCons is activated it only builds and tests those parts of the project that are out of date.
Scons documentation is online. The SCons Tutorial is useful as a starting point, but not as a reference. The SCons Man Page is a very useful reference once you have learned the basic ideas.
If all is well in your project, you should be able to run SCons at the top level and it will go and build and test everything. By default it stops as soon as it encounters an error. A successful build looks like this:
> op/scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: building associated BuildDir targets: build
g++ -o build/cpp/liveaudio/DCAudioFileRecorder.o -c build/cpp/liveaudio/DCAudioFileRecorder.cpp
g++ -o build/cpp/liveaudio/audio.o -c build/cpp/liveaudio/audio.cpp
ar rc build/cpp/liveaudio/libaudio.a build/cpp/liveaudio/audio.o build/cpp/liveaudio/DCAudioFileRecorder.o
ranlib build/cpp/liveaudio/libaudio.a
... [ for many lines ]
python -O -m onyx.onyx_py_compile build/templates/__init__.py
python -m onyx.onyx_py_compile build/templates/module.py
python build/templates/module.pyc > build/templates/module.log-doctest
verify_log_doctest( build/templates/module.log-doctest > build/templates/module.log-doctestverify )
python -O -m onyx.onyx_py_compile build/templates/module.py
summarize(["doctest_summary"], [])
cat doctest_summary
managed_modules 36 tested_modules 0 tested_statements 0 num_ok 0 num_bad 0
scons: done building targets.
Note
If the build is successful the SCons output will finish with the line scons: done building targets. If the build failed then the SCons output will end with something like scons: building terminated because of errors. or scons: done building targets (errors occurred during build).
If there is no work to be done when you run SCons you will get the following:
$ op/scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: building associated BuildDir targets: build
scons: `.' is up to date.
scons: done building targets.
Usually, when you’ve modified code, some subset of the code will get rebuilt and retested.
As you can see from the log, SCons is configured so that all the output of the build process goes into the build/ subdirectory in the project. Within this subdirectory there will be a platform-specific subdirectory, e.g. build/darwin-posix-i386-32bit-le/. The output of the build process goes into this platform-specific subdirectory. This allows you to build from multiple platforms that share the same local branch.
To remove everything that gets built by SCons use the -c command-line option:
$ op/scons -c
When there is a problem during the build SCons stops immediately. If you’d like SCons to continue despite errors, use the -k command-line option:
$ op/scons -k
SCons is threadsafe. If you have multiple cores on your machine you can get a significant speedup of the build by telling SCons how many jobs it can run simultaneously. Empirically, we’ve observed that specifying 1.5 to 2 times as many cores as you have gives the fastest builds. E.g. on a 2-core system:
$ op/scons -j 4
Note that the outputs of multiple simultaneously running jobs will be interleaved.
To stress the build process, you can tell SCons to randomly select the order in which it will run jobs (while still respecting the underlying dependencies of the build):
$ op/scons -j 4 --random
TTD: