By default no resources can be controlled from mcollective. The feature is enabled in the mcollective agent but it has an empty whitelist. Consider this feature a really powerful shotgun. The whitelist protects you and everyone who depends upon that foot you are aiming at. Be careful.

These are the default configuration options:

plugin.puppet.resource_allow_managed_resources = true
plugin.puppet.resource_type_whitelist = none

If you want to allow resource control, you would need to edit the mcollective/server.cfg file with either a whitelist or a blacklist of resources which can be controlled.

plugin.puppet.resource_type_whitelist = package,service

plugin.puppet.resource_type_blacklist = exec

MCollective does not allow you to mix white and black lists.

By default no resource defined in the puppet catalog can be controlled from mcollective, so as to prevent mcollective from making a change against the Puppet policy. Sending alternate options for a resource in the puppet catalog is most likely to simply be overwritten the next time Puppet runs without the same options. In a worse case, well… sorry about the foot.

To allow MCollective to alter resources under Puppet’s control, enable the following setting.

plugin.puppet.resource_allow_managed_resources = true

If you are unable to match a host using the --with-class filter option, the first thing to do is get an inventory of the node with mco inventory hostname. If you find that the inventory does not list any classes for a host, then the classes.txt file that mcollectived is trying to read is not being written to by Puppet.

The classes.txt file is written out by the Puppet Agent during each run. In the [agent] section of puppet.conf is a variable classfile. This defaults to $statedir/classes.txt and $statedir defaults to $vardir/state. MCollective defaults to the same location as Puppet does on every platform.

However this variable can be overridden in both puppet.conf and mcollective/server.cfg. If you do not see Puppet classes in the output of an inventory request for a puppetized node, you should check the following two values and ensure that they match up.

$ sudo puppet apply --configprint classfile
/var/lib/puppet/state/classes.txt
$ grep classesfile /etc/mcollective/server.cfg
$ mco rpc rpcutil get_config_item item=classesfile -I heliotrope
heliotrope
   Property: classesfile
      Value: /var/lib/puppet/state/classes.txt

If the classfile from Puppet matches what is above, then mcollective doesn’t need an override in server.cfg. If any different value is found, you may want to set them explicitly to match in both files.

# /etc/puppet/puppet.conf
[agent]
  classfile = $statedir/classes.txt

# /etc/mcollective/server.cfg
classesfile = /var/lib/puppet/state/classes.txt

If you are unable to match a host using the --with-fact filter option, the first thing to do is get an inventory of the node with >mco inventory hostname. If you find that the inventory does not list any facts for a server, then the facts.yaml file that mcollectived is trying to read is not being written to by Facter or Puppet.

For MCollective to know about Facts, there needs to be a parameter named plugin.yaml defined in mcollective’s server.cfg. The value of this parameter should be a filename that lists the server’s facts in YAML format, usually /etc/mcollective/facts.yaml.

# /etc/mcollective/server.cfg
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml

The target for the plugin.yaml parameter could include multiple filenames separated by a colon in Unix systems, or a semi-colon for Windows servers. If the facts do not show up after restarting mcollectived, then the most likely problem is the formatting of the YAML within the file.

The most basic way to collect system facts was described in Facts. A more elegant and flexible solution which can use Puppet-generated facts or values was documented in Sharing Facts with Puppet. It doesn’t matter how you generate your system facts, as long as they are written in YAML format to the listed file.

Confirm that one of the following is configured to write out facts to this file.

You can install Facts plugins other than YAML from the Puppet Labs Forge, GitHub, or other repositories as discussed in Finding Community Plugins. You can also build your own as documented in Creating Other Types of Plugins.

If you are unable to match a host using the --with-identity or -I filter option, the first thing to do is confirm that mcollectived is running on the server. This is the most likely reason for a failed response by name.

The next step is to check and see what the configured identity in the server configuration might be:

$ grep identity /etc/mcollective/server.cfg
#identity=

In this situation, the identity is not hardcoded in the server configuration, so we’ll have to look elsewhere.

The default identity for the node is the output of the hostname command. If you are using Puppet, we can query Puppet for its certname, which we can use as a fact to query the node identity.

# server
$ sudo puppet apply --configprint certname
heliotrope.example.net

# client
$ mco rpc rpcutil get_config_item item=identity --wf clientcert=heliotrope.example.net

heliotrope
   Property: identity
      Value: heliotrope

Likewise, you can use any other fact or class as previously described to locate the node. For example, there are only two CentOS hosts in my testlab.

$ mco rpc rpcutil get_config_item item=identity --wf operatingsystem=CentOS

Discovering hosts using the mc method for 2 second(s) .... 2
 * [ =================================================> ] 2 / 2

geode
   Property: identity
      Value: geode

heliotrope
   Property: identity
      Value: heliotrope

Summary of Value:
        geode = 1
   heliotrope = 1

Finished processing 2 / 2 hosts in 18.38 ms

If you want an MCollective node to think of itself with a different name, then set identity in server.cfg:

server.cfg. 

identity = iambatman

If you are using configuration management like any sane person, you can have the variable set from the configuration management’s knowledge of itself. For example, here’s a Puppet template fragment to ensure the MCollective node knows itself by the Puppet certificate name, rather than the output of hostname:

server.cfg.erb. 

identity = <%= scope.lookup('::clientcert') -%>

The most common source of node name confusion is based around the use of node names or FQDNs in the hostname of a system. For example, you can set a node’s hostname to either a simple name or you can include the domain.

$ grep HOSTNAME /etc/sysconfig/network   # RedHat location
HOSTNAME=heliotrope
$ hostname
heliotrope
$ hostname -f
heliotrope.example.net

With this setup the MCollective identity was heliotrope while the Puppet certname was heliotrope.example.net. You can resolve that mismatch by changing /etc/sysconfig/network on RedHat-derived systems, or /etc/hostname on Debian-derived systems or /etc/rc.conf on *BSD systems. Or you can leave it alone, so long as you understand the difference.

The lack of matching between Puppet and MCollective does not create any explicit problems. My test setup uses short node names (e.g. “heliotrope”) for MCollective while Puppet always uses the FQDN of the host.

Absolutely nothing breaks out of the box by having different identities in Puppet and MCollective, it only affects how you might write your custom plugins. In the author’s personal opinion if you have many hosts with entirely unique hostnames across your DNS domains, you can save a lot of typing by leaving the domain name off of the hostname. Other people have different opinions drawn from their experiences. Your Mileage May Vary (YMMV).

Many of the weirder problems observed on the mailing list end up being due to the clients and servers having a different idea of what time it is. Before you take any other debugging steps, ensure that your systems have a consistent view of the time.

client$ date +%s
1396310400
server$ date +%s
1396310402

Allowing for the difference in time taken for you to run the commands on these two systems, they should be within a few seconds. In modern NTP time-sync 1/100th of a second is a considerable gap, so most systems should be easily within the same second.

The reason this is important is due to how messages are constructed. Every MCollective message sent out contains the current timestamp, and a ttl to indicate how long the message is valid.

{
  :msgtime     => 1396310400,
  :ttl         => 60,
  :filter    => {
                   "fact"   => [{:fact=>"operatingsystem", :value=>"Debian"}],
                   "agent"  => ["package"]
                 },
 :senderid    => "desktop.example.net",
 :msgtarget   => "/topic/mcollective.discovery.command",
 :agent:      => 'discovery',
 :collective' => 'mcollective',
 :body        => body,
 :hash        => "88dd360f13614b7db83616ba49deb130",
 :requestid   => "70141ca8a465954706a51ef6a7d4914e"
}

In the situation described by this packet, the request is valid from 1396310400 until 1396310460. If your server receives a request from a client too far in the past, then the request will be ignored because the TTL has already expired. Even weirder problems can occur with clients in the future, from the server’s perspective. It is absolutely essential that all of the systems in the collective have a consistent view of the time.