NAME helm - Easy server and cluster automation SYNOPSIS helm TASK [OPTIONS] helm help tasks helm help [TASK] # patch the same file on all of the machines in your cluster helm patch --file my_fix.patch --target /opt/cool_system/do_stuff.pl # run a command on only the servers which run memcache helm run --command "pidof memcached" --roles memcache # copy a file to every server in your cluster helm put --local foo.tar.gz --remote /tmp/bar.tar.gz # rsync a local folder to specific servers helm rsync_put --local lib/important --remote /tmp/important_libs --servers web1,web2,web4 DESCRIPTION "helm" is command-line utility to make it easy to automate system tasks for individual machines, a cluster of machines or a subset of machines in a cluster. It has the following features: * Combine multiple commands into a single tasks and to have groups of related tasks gathered together. * Uses SSH as the transport layer for tasks and uses SSH keys for automatic authorization. * Simple optional configuration file describing your cluster, allowing tasks to target a single machine, multiple machines which share the same role, or all machines in your cluster. Can also be extended to pull configuration from more complicated sources like LDAP, etc. * Logging of each action performed to multiple channels: console, log file, irc, email, etc. * Interact with the remote processes via "STDIN", "STDOUT" and "STDERR". * Convenient .helmrc file to reduce the number of options you need to pass on every invocation of "helm". * Locking on the client and/or server so that multiple invocations of "helm" aren't running at the same time. OPTIONS "helm" has several options that can be specified for all commands. Any remaining arguments are passed to the task that was invoked. These global options are: --servers Specifically list out the hostnames of the servers to send the task to. These can either be full hostnames (or an unambiguous abbreviation of your servers as defined in your CONFIGURATION). Multiple hostnames must me specified in a comma separated list. If no servers (or roles) are specified, then the task will be performed on all servers (as defined in your CONFIGURATION). # using full hostnames --servers 192.168.0.23,192.168.0.24,web1.company.com # using abbreviations from config file --servers web1,db2 # using abbreviations from config file and ranges --servers web[1-3],db[1-2] --roles Instead of specifying the servers explicitly, you can instead specify which server roles you want to target (as defined in your CONFIGURATION). Multiple roles must be specified in a comma separated list. If no roles (or servers) are specified, then the task will be performed on all servers (as defined in your CONFIGURATION). # single role --roles web # multiple roles --roles web,cache --exclude Specifically exclude servers from the list to be used. This can be combined with both "--servers" and "--roles" and even by itself to exclude certain servers from the list of all servers as defined in your CONFIGURATION). # combined with --servers --servers web[1-10] --exclude web3 # combined with --roles, using ranges --roles web --exclude web[4-5] --exclude-roles Specifically exclude servers with certain roles from the list to be used. This can be combined with both "--servers" and "--roles" and even by itself to exclude certain servers from the list of all servers as defined in your CONFIGURATION). # combined with --servers: any web[1-10] servers that aren't also caches --servers web[1-10] --exclude-roles cache # combined with --roles: any web servers that aren't also proxies --roles web --exclude-roles proxy --parallel Execute tasks in parallel on the remote servers. The default is to execute in serial instead. --parallel-max The maximum number of parallel processes to be running at the same time if "--parallel" is used. The default is 100. --config Which resource to use for pulling cluster configuration information (see CONFIGURATION). --config helm:///etc/helm.conf --log-level The log level used. Can be one of "debug", "info", "warn", and "error". Defaults to "info". --log Log messages to a specific channel. Multiple channels can be specified by specifying this option multiple times. The value of this option is the URI of the channel to be used. See "LOG CHANNELS" for more details. # send log messages to a comany tech IRC channel --log irc://helm@irc.company.com/tech # log messages to a file and to email address --log file:///var/log/helm.log --log mailto:helm@company.com?from=helm@company.com Log messages are also sent to "STDERR" unless the --quiet option is also passed. --load Allows you to load other 3rd party plugin modules to extend "helm" functionality. This could be to add more log channels, different configuration loading, etc. The value of this option is the full Perl module name of the plugin. This can be specified multiple times. # load a hypothetical Yammer log plugin --load Helm::Log::Channel::Yammer # load hypothetical custom LDAP configuration and --load Helm::Conf::Loader::CompanyLDAP See "WRITING HELM PLUGINS" for more information. --sudo The user that should be used to perform the commands on the remote server. The actual SSH connection will be made using the current user's SSH keys, but then once the connection is made to the remote server, it's sometimes useful for the commands to be run as a different user. We also try to make sure that things like file permissions (on tasks like "put" and "patch") are also handled so that the resulting files are owned by this sudo user. --lock This options allows you to have control over whether concurrent "helm" processes can be running on either the local or remote servers. The value can be one of: "none", "local", "remote", "both". --no-rc-file Suppress the default loading of the ".helmrc file" file. --port The port to use for SSH on all of the remote servers. Defaults to the standard (22). --timeout The timeout in seconds to give the ssh connections. Default is 30 seconds. --continue-with-errors If a command fails executing against this will normally cause execution to halt and not run against any other remote servers. Using this option will tell helm to continue on to the server(s) even if a previous one has errors. --quiet Suppress the default logging to "STDERR" --man Display this documentation --version Display the version of Helm installed --dump-config Display a dump of the configuration data as understood by Helm --debug Tell helm to dump out verbose information about what it is doing internally to a log file named helm.debug in the current working directory. This is different than the "--log-level" since that is mainly for end user logging of individual task actions. This flag is meant to debug the internals of helm, including the logging subsystem, which is why it's a separate flag. --sleep Sleep a given number of seconds in between each server. TASKS Helm comes with the following built-in tasks To get more information on how to run each task, simply use "helm help": helm help run run This is the simplest of helm commands and will run some arbitrary command on the remote server. get Get a file from a remote server and put it on the local machine. put Put a local file onto a remote server. rsync_put Put a local file onto a remote server using "rsync" rather than a full copy. patch Patch a file on a remote server with a patch on the local machine. unlock Clear out any stuck Helm locks on a remote server. CONFIGURATION By default, "helm" doesn't use a configuration file, but certain features require it (using roles, server abbreviations, etc) so it's best to have one. You can tell "helm" which configuration resource to use by using the --config option. Currently, only the "helm://" URI scheme is supported. --config helm:///etc/helm.conf A configuration file will look something like this: Role web Role db Role db_master Role db Role db_slave This configuration would define 7 servers (web1.company.com, web2.company.com, web3.company.com, web4.company.com, web5.company.com, db1.company.com and db2.company.com). It defines 4 different roles (web, db, db_master, db_slave). "helm" currently just supports a single configuration resource format ("helm://"), but the internals are flexible enough that more formats could be supported in the future, including other configuration methods like LDAP, etc. If, for instance you wanted to support a URI like: --config ldap://ldap.company.com See "WRITING HELM PLUGINS" for more information. If you are having problems getting your configuration right, you can pass the --dump-config option to tell "helm" to display what it thinks things are configured to be. LOG CHANNELS "helm" can be told to send various levels of log data to different channels. By default we support a log file, IRC and email logs. We also support various log levels (see the --log-level option). You can specify which channel is used by giving a URI which indicates what type of channel and where to send the log. The following URI schemes are supported: file:// This is basically a log file where messages are immediately appended. --log file:///var/log/helm.log irc:// NOTE: IRC logging is still experimental and quite tempermental. Improvements are welcome. This is an IRC channel where messages are immediately sent. For example to send messages to the "sysadmin" IRC channel on "irc.company.com" using the user "michael" and the password "s3cr3t" you would have: --log irc://michael@irc.company.com/sysadmin?pass=s3cr3t mailto: Similar to Mail-To links in HTML documents, this just specifies an email address to log. Log messages aren't sent immediately, but are instead queued up to be sent once the command has been completed. --log mailto:michael@company.com?from=helm@company.com Plugins can be written to load allow other log channels. See "WRITING HELM PLUGINS" for more information on how this is done. THE .helmrc FILE The .helmrc file contains command-line options that are prepended to the command line before processing. Multiple options may live on multiple lines. Lines beginning with a # are ignored. A .helmrc might look like this: # always log to a file --log file:///var/log/helm.log # always load our custom plugins --load Helm::Conf::Loader::CompanyLDAP --load Helm::Log::CompanyYammer --load Company::CustomHelmTasks "helm" looks in your home directory or the current directory for the .helmrc. You can specify another location with the "HELMRC" environment variable. If --no-rc-file is specified on the command line, the .helmrc file is ignored. WRITING HELM PLUGINS Helm can be extended in many ways to make it more convenient for your projects. Helm has 3 customization points where plugins can be written and registered to interact: "task", "log" and "configuration". Using the "--load" option, you can tell Helm about your custom or 3rd party Perl modules that you would like to load. Each plugin module must register itself with Helm along with the type of events it will handle using the "Helm->register_module" method. For example, if I were creating a new plugin module for a custom task named "spiffy" I might invoke Helm like: helm spiffy --load MyCompany::Helm::Task::spiffy And my Perl module might look something like this: package MyCompany::Helm::Task::spiffy; use strict; use warnings; use Helm; use Moose; extends 'Helm::Task'; Helm->register_module('task', spiffy => 'MyCompany::Helm::Task::spiffy') sub validate { my $self = shift; # custom validation } sub execute { my ($self, %args) = @_; # do something spiffy } 1; Most likely you'd put the "--load" statment in your .helmrc file so you wouldn't have to worry about it again. Now we'll get into the details of what is expected of each type of plugin. Task Plugins A task plugin should inherit from Helm::Task and implement the following methods: validate This method would perform any custom validation needed before the task is executed against any servers. Normally this involves validating the custom options this task might use. This method only receives a single object, the task itself. As an example, lets say you want a "--nifty" option to your "spiffy" plugin above: helm spiffy --nifty foo Then your validation method might look something like this: sub validate { my $self = shift; my $helm = $self->helm; my $extra_options = $helm->extra_options; my $nifty = $extra_options->{nifty}; $helm->die("You need to provide a --nifty option!") unless $nifty; $helm->die("--nifty option ($nifty) is not a valid value") unless $nifty =~ /^fo+/; } execute This is the meat of your task plugin and is where the work happens. This method is called once for every server the task is being executed against. It receives the following named arguments: ssh A Net::OpenSSH object with an already open SSH connection to the server in question. server A Helm::Server object for the currently executing task. As long as your method doesn't die (or preferrably calls "$helm->die()", then we will assume that all was fine and dandy in the execution of the task. Log Plugins Log plugins can offer new channels for logging Helm output based on the URI given to Helm. For instance, if you wanted to send SMS logging of critical messages only, you might invoke helm with a logger like: helm foo --log sms:+15105550101 --load MyCompany::Helm::Log::SMS And then your SMS module might look something like package MyCompany::Helm::Log::SMS; use strict; use warnings; use Helm; use Moose; extends 'Helm::Log::Channel'; Helm->register_module('log', sms => 'MyCompany::Helm::Log::SMS'); sub initialize {} sub finalize {} sub start_server {} sub end_server {} sub debug {} sub info {} sub warn {} sub error { my ($self, $msg) = @_; # send SMS message } 1; In this example, we don't care about anything except errors (since SMS messages cost money and would get really annoying for anything with frequency). Log plugins need to inherit from Helm::Log::Channel and implement the following methods: intialize finalize start_server end_server debug info warn error Configuration Plugins Configuration plugins can implement new ways to load configuration data about your servers based on the URI given to helm. For instance, if you wanted to load the list of your servers and their roles from a company LDAP server, you might invoke Helm like: helm foo --config ldap://ldap.company.com --load MyCompany::Helm::Conf::ldap Helm will look for the last module registered to handle the "ldap" scheme of that url. That module might look like: package MyCompany::Helm::Conf::ldap; use strict; use warnings; use Helm; use Moose; extends 'Helm::Conf::Loader'; Helm->register_module('configuration', ldap => 'MyCompany::Helm::Conf::ldap') sub load { my ($self, %args) = @_; my $uri = $args{uri}; # poke around in our LDAP server and create a list of Helm::Server objects my @servers = ...; # then return a Helm::Conf object return Helm::Conf->new(servers => \@servers) } 1; A configuration loading plugin should inherit from Helm::Conf::Loader and implement the following methods: load This method is the backbone of a configuration plugin. It receives the following named arguments: uri A URI object representing the URI passed on the command line to be loaded by this configuration loader. helm The "Helm" object doing the loading. This method must create a list of Helm::Server objects and use them to return a Helm::Conf object. CAVEATS Multi-Platform This has been developed and tested on Linux (with bash as the shell on the remote hosts) only. Dealing with multiple platforms and writing multi-platform tasks has been punted to the future. Alpha Software This software is very ALPHA, which means it's interface is likely to change in the future. It's used currently and happily in production but please be aware if you start using it that you'll probably want to follow future versions carefully to make sure you aren't bitten by API changes as thing are flushed out. TODO In the not too distant future, we'd like to add the following features to Helm: * Add a capture option which allows stdout/stderr to be handled differently * Add a compare option which allows the output (stdout/stderr) to be compared between servers in an intelligent manner * A real exception system to avoid parsing error messages