Perceus v1.x manual

About this document

This document is a work in progress. It is a rolling draft. If you find information that is not correct, please use the Contact us form to let us know.

About Perceus

Perceus is the next generation of enterprise and cluster provisioning toolkit. Created by the developers of Warewulf (one of the most utilized Linux cluster toolkits), Perceus redefines the limits of scalability, flexibility and simplicity. Perceus has the advantage of being planned out from inception, and with this we are able to deliver what we feel is the definitive solution for all cluster platforms regardless of their clustering needs or software budget.

The de facto freely available, stateless community solution has been Warewulf for many years now. Facilitating open customization and site-required technologies without trading scalability, ease of use or simplicity, Perceus maintains this tradition and even takes it further. Capable of the same proven stateless node provisioning ideologies, we are now also able to easily implement other provisioning methods for greater flexibility (e.g. stateful and hybridized solutions).

What is perceus?

Perceus is the next generation cluster and enterprise tool kit for the deployment, provisioning, and management groups of servers. Employing the power of the Perceus OS and framework, the user can quickly purpose a machine out of the box. Perceus truly makes a computer a commodity, allowing an organization to manage machines in a scalable fashion.

Perspectives

Computer Perspective: The computer is identified solely by its MAC address; it knows nothing about itself: what OS it should run, what hostname it should have, or what IP address it should be given. The machines introduces himself through PXE, telling Perceus its MAC address.

Perceus Perspective: Once Perceus knows about the MAC address, it has an established relationship with the computer. Based on the submitted MAC address or identity, Perceus issues the computer an OS, hostname, optional IP addresses, etc. It can also issue other customizable parameters such as nfs mounts, runscripts, and /etc/sysctl.conf.

The mechanism:

Perceus functions like a two stage rocket. In the first stage, the computer is booted into the Perceus OS. The Perceus OS interacts with Perceus daemon on the master node to make preparations for the second (and final) OS. Perceus downloads, configures, and customizes the image of this second OS. At this stage, Perceus can set the hostname and the IP address and perform other customizations. When the preparations are complete, Perceus enters the second stage by spawning the second OS through kexec.

Flexibility

By expanding on the proven Virtual Node File System (VNFS) technology invented by Warewulf, Perceus can provision hardware from bare metal with any of the premade VNFS capsules or a custom VNFS capsule that you can create for it. Some of the features include:

And since it's open source, you can customize Perceus for either HPC applications or commercial data center requirements.

System Requirements

Architectures

Supported architectures are x86 (ia32) and x86_64. There have been people that have ported Perceus to other platforms, but do this at your own risk. If you do however successfully do this, please share your notes with us upstream. ;)

We have had requests to support ia64, and PPC64 as well and we will do so when hardware and development resources become available.

Supported Operating Systems

While there are plans on supporting other Operating Systems, presently Perceus is a Linux solution. Even more specifically, Perceus is developed and primarily tested on the following operating systems:

Other distributions that should work but are not typically verified are:

Your mileage may vary for each of these.

Node requirements

There are several layers of hardware support that is needed for Perceus to function correctly. First the system must support PXE (network boot subsystem). Then the Perceus initial ram file system (initramfs) itself must support the hardware. Lastly, the provisioned VNFS must also support the hardware.

Perceus usually works with most hardware configurations but if you have a problem, send a detailed email to the maintainers at the Contact Us form.

Building the ideal cluster architecture

The Cluster Network

Perceus takes a typical cluster provisioning approach to networking. This being a master node to slave node relationship. This relationship is networked by one or more private networks that is used for DHCP, provisioning and administration of the nodes (among other non-Perceus related things).

Simplest Form

The master node is connected to 2 networks. One being publicly accessible, the other being private and shared specifically only to the nodes and devices that are accessible to the cluster itself (e.g. a network attached storage device).

Literally the master node is connected to the public LAN/WAN on one port, and the next port is connected to a standalone switch or set of switches to which all nodes are plugged into.

Larger scale solution

Starting with the simplest form, you can have the Perceus master(s) and all nodes on a single private network with a dedicated firewall/gateway bridging this network to the LAN/WAN with either some gateway login system bridging the gap, or doing a port forward/masquerade from the firewall to a login system on the private network.

Multiple Networks

Once the cluster network has been correctly configured, it is possible to multi-home the nodes (two or more network controllers in each node connected to additional networks).

A typical example here would be if each of the nodes must have real publicly accessible IP addresses.

System Roles

The role of the Perceus server

There are several ways to run Perceus to make it scale for the specific cluster size that it will manage. You can always redefine any of these methods later, but it will require more hackery.

Single Perceus master, local state directory

Running Perceus like this is the most straight forward and the default out of the box capable solution. It basically configures Perceus to utilize a local state directory.

In this configuration, the Perceus master will need to either export out this directory to the nodes via NFS (default), or you can utilize Apache to distribute the VNFS content (not default and less clean, but due to religion and specific needs this option exists).

How many nodes this method supports is very dependent of the capabilities of the hardware configuration of the Perceus master. Theoretically a decent hardware configuration can support about 32 simultaneously booting nodes (this is not the total number of supported nodes, rather how many nodes can be provisioned in parallel). Using a lightweight VNFS capsule, this makes provisioning a 512 node cluster possible in about 4 minutes. As long as you stagger the boots, this model will scale into the thousands of nodes.

This method is subject to variances in addition to hardware configuration as well. NFS implementation, specific NFS and kernel tuning, etc..

Single Perceus master, non-local state directory

Similar to the above, the primary difference here is that another system is used for NFS. This could be either another Unix/Linux system or something like a NAS device or even a parallel file system. This can drastically increase the number of nodes that can be provisioned simultaneously with ranges highly dependent on the NAS storage system used.

Multiple Perceus master, non-local state directory

Again similar to the above example, it is possible to run multiple Perceus masters all mounting the non-local state directory thus sharing all of the state and provisioning load.

Definitions and terms used throughout this document

VNFS

The Virtual Node File System began as a simple chroot in the early Warewulf days. As Warewulf and the idea of the Virtual Node chroot (or file system) matured, the importance of the VNFS became apparent. Warewulf utilized scripts to create and do initial configuration of VNFS images which was nice from the distribution perspectives, but caused problems with reproducibility and cross platform VNFS creations.

Another major limitation with Warewulf's implementation of the VNFS was that the kernel was inherently separate from the VNFS image. Meaning, that Warewulf would utilize the kernel from the master's underlying OS (thus the kernel you want to boot your nodes must be installed on the master). This model seems logical at first, but quickly limits the combinations of distribution/kernel relationships that can be maintained.

Introduction of the VNFS capsule

Different from the VNFS image (which is basically just a chroot) Perceus introduced the VNFS capsule. The capsule not only contains the chroot image, but also all of the necessary provision scripts, kernel and kernel modules, and other configuration tools. This means that the VNFS itself controls its own methods for provisioning, setup, hardware support, application support, etc.

The VNFS capsule allows us now to maintain what goes on the node file systems, offer support models for that, and allow us to delivery complete solutions.

Provisionary State

Nodes contact the Perceus daemon from a variety of states depending on what the node is doing at that time. For example, the init state is hard coded in Perceus and it tells the Perceus daemon that the system is now ready to be provisioned. Another provisionary state that is used (but not hard coded) is the ready state.

The provisionary state is communicated to the Perceus daemon via the Perceus client daemon which will run on the nodes inparticular. The client will contact the master with its provisionary state, and the master will send the client what it should do in that state.

The module command makes use of the provisionary state in the form of a heariachy. For example, init/all means all nodes will run that module when they are provisioned. On the contrary, init/group/foo means that only nodes part of group foo will run this module.

Installation

If Perceus came with the installation of your Operating System then you can skip this section.

Perceus is capable of getting installed using very standard GNU type build methods or via the systems package manager (RPM currently supported).

Before you install

Local or Remote State?

Before you install Perceus, you should make the decision and take any required steps for what role you want Perceus to take. For example, if you decide to utilize a remote state directory, then you should mount the state directory at this point (you can do it later, but then you have to copy the data from the local directory to the remote mount point and then mount... possible, just a PITA).

For example, if your remote NFS server has the IP address 10.0.0.5 and the exported Perceus state directory is at /vol/perceus, then you would need to add that directory to the master's /etc/fstab. Assuming that you want to put your local state directory at /var/lib/perceus, you can add the following line to your /etc/fstab:

10.0.0.5:/vol/perceus   /var/lib/perceus        nfs     defaults        1 2

Then you should run the following commands to mount the file system:

# mkdir -p /var/lib/perceus
# mount /var/lib/perceus

Installing Dependencies

Prerequisites to Perceus

Perceus requires some applications and services to be provided by the installed operating system itself. These may have different names on the various distributions of Linux, but the upstream packages names are as follows:

  • NFS
  • Perl
  • Perl-Unix-Syslog
  • Perl-IO-Interface

There isn't a need to configure any of the services once they have been installed, Perceus will prompt to do this for you.

If you would like to customize any of the services (e.g. DHCP) you will not break Perceus as long as you configure the PXE support properly.

Some of the dependencies are redistributed from the Perceus download page and can be found at http://www.perceus.org/downloads/ in the version/dependencies directory (e.g. perl-Unix-Syslog, perl-Net-ARP, etc...).

Building and Installing Perceus

Assuming that your distribution of Linux doesn't offer Perceus as a supported package you will need to install Perceus either by RPM or source tarball.

Building/Installing from source tarball

 # tar xvzf perceus-1.3.0.tar.gz
 # cd perceus-1.3.0
 # ./configure --prefix=/usr/local/perceus/`cat VERSION`/ --localstatedir=/var --sysconfdir=/etc
 # make
 # make install

This will install Perceus into /usr/local/perceus/[version] except for the localstatedir and sysconfdir which would be in the real system. The configure line should be tweaked for personal preference.

Building/Installing RPMS from source tarball

 # Some versions of tar need this option for RPM to build properly from tarball
 export TAR_OPTIONS=--wildcards
 # rpmbuild -ta perceus-1.3.0.tar.gz
 Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.68349
 + umask 022
 + cd /usr/src/rpm/BUILD
 + LANG=C
 + export LANG
 + unset DISPLAY
 + cd /usr/src/rpm/BUILD
 ...
 Wrote: /usr/src/rpm/SRPMS/perceus-1.3.0-1.src.rpm
 Wrote: /usr/src/rpm/RPMS/x86_64/perceus-1.3.0-1.x86_64.rpm
 Wrote: /usr/src/rpm/RPMS/x86_64/perceus-provisiond-1.3.0-1.x86_64.rpm
 # rpm -ivh /usr/src/rpm/RPMS/x86_64/perceus-1.3.0-1.x86_64.rpm

This will create RPMS that can be used not only on this system, but other systems running the same distribution version.

Perceus Startup

Getting to know Perceus

Perceus stores its configuration in several places on the file system. This is to allow for redundant and load balanced Perceus configurations.

The primary configuration for Perceus itself is hard coded to /etc/perceus (even if you choose a different --prefix and/or --sysconfdir). Inside this directory, Perceus will expect to find several configuration components all of which are specific to the Perceus "instance" on that system. I used the word instance because depending on your configuration, you may have multiple Perceus masters.

The node specific configurations will get put into a directory that is dependent on how Perceus was installed. If you specified the --localstatedir option to the configure script, then it will be put under there (e.g. --localstatedir=/usr/var would set the Perceus state directory to /usr/var/lib/perceus). Inside the Perceus state directory, there is a directory called database/ (as of Perceus 1.2). You should never need to touch this directory by hand.

Perceus modules, VNFS capsules and node scripts are also stored in this state directory, and you will find links to the installed VNFS capsules and the node scripts in /etc/perceus for convenience.

Lastly modules that require configuration files should default to creating their configuration files in /etc/perceus/modules, but this is at the control of the module author.

Configuration

The Perceus configuration file

Before starting up Perceus for the first time, it is necessary to edit the config file located at /etc/perceus/perceus.conf. The settings are dependent on the role of this master.

note: See role page for definitions and explanation.

Using NFS Vs. HTTP transfer method

You can utilize either NFS or HTTP for provisioning the VNFS image on the nodes. There are some advantages and disadvantages to both and if you are not sure what to use, I recommend to stick to the Perceus default mechanism which is NFS. It is easier to implement and allows for greater functionality. If you also choose anything aside from the default local state, single Perceus master role it will require some trickery.

This documentation for the most part will assume the default vnfs transfer method NFS configuration.

Examples

local state directory: The most straight forward way of implementing Perceus is to use the local state model. For this you will just need to set the vnfs transfer master to the IP address of the local network controller facing the nodes.

For example, if my public network device is eth0, and cluster network device is eth1 with an address of 10.0.0.1/255.255.0.0, then I will set the following:

# Define the IP Address of the network file server
vnfs transfer master = 10.0.0.1

note: When you run Perceus for the first time, it will notice that this address is local, and will offer to configure NFS exports for you

Remotely mounted state directory: Here you will need to set two configuration options. One to set the IP address of the vnfs transfer master, the second to set the path or prefix on the remote server to the perceus state directory.

For example, if my remote NFS server is exporting the Perceus state information at 10.0.0.5:/vol/perceus, I will need to set the following:

# Define the IP Address of the network file server
vnfs transfer master = 10.0.0.5

# Define the VNFS transfer location if it is different from the default
# ('statedir'). This gets used differently for different transfer methods
# (e.g. NFS this replaces the path to statedir, while with http it is gets
# prepended to the "/perceus" path).
vnfs transfer prefix = /vol/perceus

In this case, you will need to confirm that you have mounted the remote Perceus state directory at wherever you configured your --localstatedir + lib/perceus (e.g. /usr/var/lib/perceus).

Initialization

Running Perceus for the first time

The first time Perceus is run it will take you through an initialization process.

Registration

Perceus will first attempt to register your installation. The purpose of registration is simply to let the authors know who is using the Perceus which in turn will be used to help obtain funding, presentation statistics, and general curiosity of the authors.

Under no circumstances will any information be shared to anyone outside of our development group without explicit approval.

DHCP configuration

Perceus will next prompt the installer for the ethernet device that is configured on the private cluster network. Do not make a mistake on this otherwise Perceus will start trying to configure nodes and assigning DHCP addresses on your public network which could lead to many systems on that network not having any network connectivity.

From the ethernet device, Perceus will obtain the network configuration and then prompt for the DHCP range that should be used for booting nodes. (while it is possible to statically address all of the nodes, new nodes require a range to boot from).

Non interactive configurations

The rest of the initialization is non-interactive but you need to pay attention to the output to make sure there are no critical errors that would cause breakage.

Exporting NFS file systems

Perceus will automatically figure out if any NFS file systems need to be exported based on your perceus vnfs master configuration. It will add the appropriate configuration to /etc/exports and then restart the NFS service if necessary.

SSH public/private keys

Many node capsules (at least the ones that can be built or distributed via Perceus directly) will automatically allow for root access via SSH keys. Perceus will create separate ssh keys specific for Perceus thus they will not interfere with any ssh keys that you already have configured.

Configuring the node hardware

PXE

Preboot Execution Environment (PXE) is a solution for bootstrapping a computer to boot off of a network interface card (NIC) rather then a local storage device. It has been standardized on by most NIC vendors and is used by Perceus to boot and provision nodes.

PXE became very prominent and considered standard for systems since about 2002. If your nodes are older then this, it is possible that they don't support PXE. It may still be possible to use Perceus utilizing Etherboot, but that hasn't been documented (yet).

Enabling PXE is different for many systems depending on their BIOS version, revision and hardware stack. In general you will want to enter the BIOS (usually pressing [F1], [F2], [F10], or [DEL] immediately at bootup). You may need to consult the motherboard manufacturer's user manual for details. Once at the BIOS configuration menus you will need to navigate to the boot order screen, and put the PXE devices above the local disk drives and under the floppy and CDROM (so you can boot locally for debugging if you need).

Keyboard errors

If you do not have a keyboard connected to the node you will want to ensure that the system will not error out on the keyboard probe. You will have to navigate a bit to find the configuration option but it will be well worth your time.

Hyperthreading

Most newer Intel CPU's offer a feature called Hyperthreading. It is a way of improving processor performance by doing threading at the hardware level and virtualizing multiple processors per core. Depending on the nature of the jobs to be run on this cluster, you may wish to disable this feature (for example this is detrimental for most HPC jobs).

Using Perceus

Congratulations!

You have successfully installed Perceus at this point. Now via normal utilization we will see how well you did.

After installation you will find the Perceus directory structure to be as was specified by the configure script. For example, if Perceus was configured with the following command:

# ./configure --prefix=/usr/perceus --localstatedir=/var

Then we would end up with the following directory structure:

If your configuration of the bin/ dir is not in a standard path, then you should add it to your PATH environment variable (not required, but makes things easier).

note: RPM package installs of Perceus will install Perceus directly into the operating system's file system (e.g. /usr/lib/, /usr/bin/, /usr/share/, etc...).

Default configurations for new nodes

New node defaults

As nodes are either added or booted into Perceus, they will be assigned the values defined in /etc/perceus/defaults.conf. Changing this file will only affect new nodes that are added to Perceus (not existing nodes)!

Node Name

The node name configuration option tells Perceus what format new node names should have. By default this is set to n####.new. The node name (just like a host name) is broken into 2 components separated by a dot. The first component is necessary as it will be the system's hostname. The second part is the cluster or subcluster name and is optional.

Names are enumerated in the range symbolized by the '#' marks. In the above example, Perceus will use 4 integers to represent the node number. When new nodes are added to the cluster, the node number will be set to the first available number in sequence. Thus if there is a gap between, n0008 and n0011 the next node will be n0009, the following will be n0010, and after that would be the next available over n0011.

The node name is the primary method for accessing the node and will be used as the system's hostname as well. Here is an example on using the node name to set a paramater:

# perceus node set enabled 0 [node name]

note: This is just a foreshadowing as Perceus commands have not been described yet. Specifics on Perceus commands and usage to come shortly.

Group Name

Node groups are handled like an extra identifier for addressing nodes. There is no separate configuration file for node groups, rather think of them as a way of operating on multiple nodes in bulk.

This means I can do something like:

# perceus group set enabled 0 [group name]

which would disable all nodes in the mentioned group.

Vnfs Name

If you have already imported a VNFS capsule into Perceus, you can set new nodes to automatically be configured to use it. The string here should be exactly the same as the output of the following command:

# perceus vnfs list

Booting and adding nodes

Node Identifiers (NodeID)

Nodes are "known" to Perceus as the network card's MAC address that the node boots on. If a node requires a new motherboard or NIC, it is very possible that the node will be unknown to Perceus and thus will be added as a new node.

Booting nodes

At this point, Perceus is ready to start adding and booting nodes. As long as the nodes have been configured correctly to PXE boot and the network is correctly setup you can just turn on the nodes in the order that you want them configured in Perceus.

When the node boots, it will come up and connect to the Perceus daemon running on the master node that supplied the DHCP configuration.

If the node is not known Perceus will automatically add it to the database and assign it the default values as specified by /etc/perceus/defaults.conf.

Adding nodes at the command line

Nodes are identified by the IPv4 MAC address of the connecting network device. If you have the MAC address of the nodes you can import it directly into the Perceus database by using the command line:

# perceus node add [MAC ADDRESS] (optional hostname)

Using a simple script you can easily import any number of nodes into the database.

Adding nodes via "other" ways

There are several scripts that are included with Perceus to facilitate adding nodes. They can be found in the $prefix/share/perceus directory.
  • import-isc-dhcpd.pl: This script tries to read a dhcpd.conf file that has static node addresses defined of the following format:

  host n0000 {
     hardware ethernet 00:00:00:00:00:00;
     fixed-address 10.0.1.0;
  }

  • insert-ethers.pl: This will scan syslog watching for DHCP requests, and immediately add any request to be a new node. While this is dangerous to run because it blindly adds nodes, it is significantly faster for booting a lot of nodes.
  • import-warewulf-2.6.pl: This will import nodes from a Warewulf node directory.
  • import-warewulf-2.4.pl: This will import nodes from a Warewulf node configuration file.

note: These scripts are being added still, script presence maybe dependent on Perceus version.

The Perceus command line reference

Perceus command line summary

For the most part (several minor exceptions), your primary interaction with Perceus will be via the command line. The Perceus command line tool is capable of managing and administrating just about all aspects of the Perceus realm.

Generally speaking the format is:

perceus [command] [subcommand] (arguments...)

The following valid commands will be described later:

  • node: Set, display and list node attributes and statistics
  • group: Set, display and list group attributes and statistics
  • vnfs: Import, delete, modify, sync VNFS capsules
  • module: Import, activate, display Perceus modules
  • info: Print various aspects about Perceus
  • configure: Have Perceus configure required daemons
  • contact: Perceus is capable of contacting the developers and maintainers directly.

Version drift of documentation

This documentation was written against the Perceus 1.2 release. If you are running a different version, it is possible there are some differences. This is to be expected, and if you find something, please use the Contact Us link to send us a message and let us know.

Using the 'node' command

# perceus node add [MAC ADDR] (optional node name)

This command can be used to import a new node entry into Perceus. The MAC address is a required parameter and you can optionally add the desired node name. This can be used to easily read in from a list of MAC addresses to pre-configure all nodes before they are even booted (assuming that you have the list of MAC addresses).

Here is an example script that will import (in order) all of the MAC addresses contained in the list:

# cat mac-addresses.txt | while read i; do perceus node add $i; done

You can get as creative with this as you want.

# perceus node delete [nodes]

This command will remove nodes from Perceus permanently. It takes as an argument a list of nodes, a node range, or a globing. For example:

# perceus node delete n0001 n00[05-20] n0*00 *00*0

note: When using the wild card in globing, you must take care the contents of your underlying directory because they shell itself will try to expand it. So if you have a file called "n0hello_world00" in your current directory, that is what will get passed to Perceus. If you want to glob for all nodes, then you must prefix the astrix with a backslash '\*'.

# perceus node list

Simply outputs a list of the configured nodes to standard output.

# perceus node replace [old node name] [new node name]

This command is very helpful when the MAC address of a node has changed (caused by changing hardware).

Commonly if a node needs to be replaced, you can simply remove the old hardware, put the new hardware in its place, and turn it on. Perceus will immediately add your new node to the database at which time you can now "replace" your old node with the new one.

# perceus node set [paramater] [value] [nodes]

This is how you will set most paramaters after a node has been initially configured (via the new node defaults). Valid values to configure are:
  • hostname: This is the node's name
  • groupname: Define the group's name
  • vnfs: Specifies which VNFS capsule this node will use
  • enabled: Enables/disables this node (disabled nodes will wait indefinably at the Perceus boot screen waiting to boot).
  • debug: If you have any issues with provisioning nodes, this will run all scripts in debug mode (set -x)
  • desc: Allows you to set a freeform description to nodes

# perceus node show (optional node list)

Dump all database fields to standard output for the user to parse through. For example:

# perceus node show | grep -i nodeid | head -n 3
n0000:NodeID=00:13:72:FB:5B:07
n0001:NodeID=00:13:72:FB:5F:B4
n0002:NodeID=00:13:72:FB:5E:C3

# perceus node status (optional node list)

Display the current status of node provisioning and activity. This will output things like hostname, provisioning state, last known IP address, and last contact time.

# perceus node status | head -n 4
HostName             Status         IpAddr                    Last Contact
n0000                ready          10.0.3.0                  00:02:39
n0001                ready          10.0.3.1                  00:02:16
n0002                ready          10.0.3.2                  00:02:15

# perceus node summary (optional node list)

Summarize the current node configuration in a tidy output. The primary difference between summary and status is that status shows the current access while summary shows node configuration information.

# perceus node summary | head -n 4
HostName             GroupName    Enabled   Vnfs
n0000                esd01            yes   caos-3.0-stateless.26.x86_64
n0001                esd01            yes   centos-4.4-stateless.31.x86_64
n0002                esd01            yes   centos-5.0-stateless.10.x86_64

Accessing by NodeID/MAC instead of by hostname

Using the -i flag to Perceus, you can identify nodes by their MAC (NodeID) address instead of by hostname. This is very important if you have multiple nodes with the same node name.

Using the 'group' command

Relationship between node and group commands

The Perceus group command works much like the node command with several exceptions. First and most obviously is that it will operate on group names rather then node or host names. The second is that there is 1 additional subcommand.

For example:

# perceus group set groupname newgroup oldgroup1 oldgroup2

which will merge all nodes in oldgroup1 and oldgroup2 into a single new group called newgroup.

# perceus group nodelist (optional group list)

This command will output the list of nodes associated with each group in an indented format.

Using the 'vnfs' command

# perceus vnfs import [/path/to/file.vnfs]

Capsules are imported into Perceus using this command. Embedded inside the VNFS capsule file itself is everything that Perceus needs to provision nodes with that capsule. It contains not only the root file system, but also the kernel, kernel modules, init and provisioning scripts, etc...

# perceus vnfs delete [vnfs name]

Remove VNFS capsules from Perceus. Caution, this command is obviously irreversible! Once a capsule has been removed, you better have a backup if you need it!

# perceus vnfs list

Display a list of all installed VNFS capsules.

# perceus vnfs mount [vnfs name]

Depending on the VNFS capsule itself, this may do different things. Generally speaking it will put the VNFS root image into a mount point or a directory structure which can be easily modified. This structure will be either placed or mounted at /mnt/[vnfs name] for easy and standardized access.

By default, mounting a VNFS image is a blocking operating meaning that pending nodes booting will wait for the capsule to be unmounted before booting. This can be different however depending on the capsule.

# perceus vnfs umount [vnfs name]

Like the above mentioned mount, this will generally unmount the capsule itself and then rebuild the VNFS image that the nodes will use (if necessary for that capsule).

# perceus vnfs close [vnfs name]

This will close a mounted VNFS image without rebuilding the image itself. This can be dangerous and could cause version drift so use this feature carefully!

# perceus vnfs configure [vnfs name]

This is done automatically when a VNFS has been imported into Perceus. You can run it again manually if for some reason it is needed.

# perceus vnfs clone [vnfs name] [copy of vnfs name]

This command will clone or make a copy of an installed VNFS capsule.

It is good practice never to operate or make major changes to production capsules. This gives you the capability of versioning your VNFS capsules.

# perceus vnfs livesync [vnfs name] (optional node list)

Livesync will update node file systems without rebooting the node. This is dangerous, but a very handy option!

If used without the optional node list, it will sync every node configured to use the specified VNFS capsule in a serial fashion. It is recommended to first sync 1 test node, then to update the rest.

Using the 'module' command

Modules and provisionary states

Perceus modules use provisionary states to determine when to run particular activated modules. The allowed provisionary states are in the following format:
  • [state]/all
  • [state]/group/*
  • [state]/vnfs/*
  • [state]/node/*

Using this model, you can be as unspecific or granular as needed to activate a module for any nodes.

# perceus module import [/path/to/module/file.pmod]

Perceus comes with some modules already imported for you, but you could optionally bring in other modules to do specific things. Once the module has been imported into Perceus, you will then have to activate that module.

# perceus module list

Display the list of currently installed modules.

# perceus module summary (optional module list)

List all of the installed modules and also specify what provisionary states the module is activated in.

# perceus module activate [module name] (optional provisionary state)

Activating a module is the same as enabling it (modules that are not set active won't do anything). Modules should come with a default provisionary state if one (or more) are not specified at the command line.

= # perceus module deactivate [module name] (optional provisionary state)

This will deactivate an activated module in the given provisionary state(s). If no optional states are given, it will deactivate in all currently activated states!

Using the 'info' command

# perceus info about

Print the about summary for Perceus. This is the same screen that gets displayed if you don't type any arguments to Perceus.

# perceus info config [configuration paramater]

This command will dump particular pieces of information to standard output. For example:

 # perceus info config prefix
 /usr/perceus/1.2
 # perceus info config "vnfs transfer master"
 10.0.0.1
 # perceus info config statedir
 /var/lib/perceus/

# perceus info system

A very useful command for generating a debug'ing starting point. This will dump information about Perceus and the master node itself.

Using the 'configure' command

Perceus Initialization

When Perceus is initialized for the first time, it will run through all of these options by default. Using these commands you can reconfigure these services, but this is usually not necessary.

# perceus configure apache

This is useful for systems using the http vnfs transfer method. It will basically put a default configuration file at /etc/httpd/conf.d/perceus.conf (default location for several distributions Apache package) which will configure everything correctly for a default install.

# perceus configure dhcpd

This will configure the DHCP server. It will take you through several interactive questions to setup the default configuration.

# perceus configure nfs

If you are running a local NFS server (e.g. vnfs transfer master is a local IP address) this command will update the /etc/exports file if needed and restart NFS.

# perceus configure sshkeys

Recreate the ssh keys used by Perceus. Caution, on existing imported VNFS capsules, this might cause you to loose password-less root ssh access!

Using the 'contact' command

# perceus contact support

This is used to email a support ticket to the Perceus developers. Doing this includes Perceus version, configuration summary and system overview which helps in remote debugging. It will interactively walk the person through a short series of questions which will help identify the user and problem. Expect a return email in 12-24 hours.

# perceus contact register

If for some reason you wish to re-register and let us know how your doing, this command will do that. ;)

VNFS Administration

Obtaining the VNFS capsule

Downloading a VNFS Capsule

You can download premade VNFS capsules supporting multiple distributions, hardware configurations, and even application stacks from http://www.perceus.org. At the time of this writing we are developing our model for distributing VNFS Capsules, and should have a reasonable selection up shortly.

Supported, tuned, and commercially available capsules

VNFS capsules are made available for the commercial distributions, high performance interconnects, tuned MPI stacks, web cluster solutions, among other solutions by Infiscale, Inc.. These capsules can be purchased and supported to fit in to Perceus with minimal work and overhead. Contact Infiscale, Inc. for more information.

Creating a VNFS capsule

At present (version 1.0.6) it is possible to create a Centos-4 VNFS capsule from scratch using the scripts that come with Perceus. Assuming that Perceus was installed into a prefix of /usr/local, here are the commands to create a stateless hybrid (tmpfs+nfs) VNFS capsule for Centos-4.4:

 # /usr/local/share/perceus/centos4-genchroot.sh
 Building in: /var/tmp/vnfs/centos-4.x86_64/
 
 ...
 
 # cd /usr/local/share/perceus
 # sh ./chroot2stateless.sh /var/tmp/vnfs/centos-4.x86_64 /root/centos-4.4.x86_64-stateless-1.0.vnfs
 Creating VNFS capsule 'centos-4.4.x86_64-stateless-1.0.vnfs'
 Moving /sbin/hotplug out of the way...
 Building vnfs.img ...
 Setting kernel: vmlinuz-2.6.9-42.ELsmp
 Creating VNFS capsule scripts
 Compressing capsule ...
 
 WROTE: /root/centos-4.4.x86_64-stateless-1.0.vnfs

Importing the VNFS capsule into Perceus

Once you have obtained the VNFS capsule you will need to import this into Perceus. Here is an example of a VNFS import:

 # perceus vnfs import caos-3.0-stateless-1.x86_64.vnfs 
 Importing 'caos-3.0-stateless-1.x86_64'
 Configuring VNFS...
 Mounting VNFS 'caos-3.0-stateless-1.x86_64'...
 The VNFS can be found at: /mnt/caos-3.0-stateless-1.x86_64
 Enter the IP address of your syslogd host (10.0.0.1):
 Creating ssh keys... rsa1 rsa dsa 
 Copying ssh public keys
 Un-mounting VNFS 'caos-3.0-stateless-1.x86_64'...
 # perceus vnfs list 
 caos-3.0-stateless-1.x86_64

Making changes to a VNFS capsule

Mounting the VNFS image

To make changes to the VNFS you first need to mount the capsule via:

 # perceus vnfs mount [vnfs name]

Once the capsule is mounted, the root file system should be available at /mnt/[vnfs name]/ (but the specific location may differ depending on the type and who made the capsule you are mounting.

Unmounting the VNFS

To activate the changes you must un-mount the VNFS using the command:

 # perceus vnfs umount [vnfs name]

Syncing the nodes

The correct way to synchronize a node is to reboot it and let it get re-provisioned. This is the best way to ensure that you don't end up with any fragments or state left over from the previous provisioned state. There is however a shortcut...

Previously with Warewulf, there was a method for updating the nodes without rebooting, but it was undocumented because it was dangerous. It is still dangerous, but now it is documented to be dangerous. This method is called livesync.

Putting the dangerous of overwriting open file handles and inadvertently resetting configuration files aside, you can use this command like:

 # perceus vnfs livesync [vnfs name] (optional node list)

note: Not all VNFS capsules will have this option accessible.

Hybridization

Hybridization is where particular chunks (in some cases the majority of the operating system) is actually redirected to a network file system via links. For example, lets say you don't want to populate /usr/share in RAM on all of the nodes, you can hybridize that directory so it will become a link that will point to a non RAM mounted file system.

The end result is what looks to be a contiguous standard operating system, but offloading parts of the operating system to a non-local media.

You would configure what files get hybridized in /etc/perceus/vnfs/[VNFS NAME]/hybridize (if the capsule type supports it). Once you edit which files or directories should be hybridized, you will need to remount and umount that capsule to activate the changes.

WARNING: Do not hybridize a directory that you plan on NFS mounting, or the directory where you configured the perceus --localstatedir during the build (if built via RPM, this is /var).

The Perceus Network Manager

The Perceus Network Manager

Perceus integrates a network management daemon (using dnsmasq) that manages several core network services. Now Perceus is tightly integrated with DNS, DHCP and TFTP services.

Perceus DNS Implementation

To use this, you can simply point each systems /etc/resolv.conf to the IP address of the Perceus master node. This will enable all nodes to be able to resolve the hostname of all nodes (even nodes that have booted via the default DHCP range). It is recommended to also add localhost to the master's /etc/resolv.conf above the other nameserver entires. This way you can always resolve dynamically assigned nodes. The entry should look like:

 nameserver 127.0.0.1

The DNS service will also proxy the DNS requests to any non localhost (127.0.0.1) nameservers specified in /etc/resolv.conf.

note: The nodes will automatically be configured via DHCP to configure the appropriate nameserver

Perceus DHCP Implementation

Chances are you have already configured the DHCP server via the Perceus initialization. By default it offers a range of addresses that it will use to boot new and non-statically configured nodes. Depending on your node requirements, it might be better to configure a static IP address for each node.

To configure static IP addresses for the nodes, simply add an entry in /etc/hosts using the exact node name as configured in Perceus. Once you have added the appropriate entries to /etc/hosts you just need to tell the Perceus Network Manager to reload:

 # /etc/init.d/perceus reload
 Sending HUP to perceus-dnsmasq...

At this point you can either reboot the node, or wait for it to renew its lease and get the new IP address.

Miscellaneous Features

  • Round robin DNS: If you add multiple lines in /etc/hosts specifying the same hostname, this will automatically enable round robin support. This is typically used for spreading out a load to multiple hosts.

Perceus Modules

Importing modules

When you install Perceus it will come with some core level modules already integrated but are not set active. Any additional modules you wish to download and import should be done with the command:

 # perceus module import [/path/to/module.pmod]

Module Activation

When a node boots and connects to the Perceus service, it will start the communication by stating what provisionary state it is in (e.g. init, boot, ready, etc...) and its NodeID. With this information, the Perceus daemon can figure out what node/boot scripts need to be run for this node.

Modules can be activated for various provisionary states (by default there is 'init' and 'ready' but this can be customized). Inside each of the provisionary states there are 4 substates (all, group, vnfs, node).

These can be used in the following form:

 # perceus module activate hostfile
 # perceus module activate hostname init/all
 # perceus module activate passwdfile init/group/newgroup
 # perceus module activate groupfile init/node/node0000 init/group/anothergroup

Module summary

You can view the current module stack using perceus module summary:

 # perceus module summary
 groupfile:
         init/group/anothergroup
         init/node/node0000
 hostfile:
         init/all
 hostname:
         init/all
 passwdfile:
         init/group/newgroup

Module deactivate

Disabling modules in particular provisionary states can be done as follows:

 # perceus module deactivate groupfile init/group/anothergroup

The Groupfile Module

About the groupfile module

Once you have activated the groupfile module, you will need to configure what groups get synchronized to the specified nodes. Activation of the module will create a directory structure in /etc/perceus/modules/groupfile where you can customize what nodes get what files. Each file is a portion of the entire /etc/group file that the node will receive.

The structure is of the format:

 /etc/perceus/modules/groupfile/
      all
      group/*
      vnfs/*
      node/*

The all file is the default and starting point that all nodes will get. From there you can edit the other files in the directory structure to fine tune which nodes get what groups.

The Passwdfile Module

About the passwdfile module

Once you have activated the passwdfile module, you will need to configure what groups get synchronized to the specified nodes. Activation of the module will create a directory structure in /etc/perceus/modules/passwdfile where you can customize what nodes get what files. Each file is a portion of the entire /etc/passwd file that the node will receive.

The structure is of the format:

 /etc/perceus/modules/passwdfile/
      all
      group/*
      vnfs/*
      node/*

The all file is the default and starting point that all nodes will get. From there you can edit the other files in the directory structure to fine tune which nodes get what passwd entries.