User authentication is the process of validating the user and thus preventing unauthorized access to the Elastic Cluster. In the X-Pack security module, the authentication process is handled by one or more authentication services called realms. The Security module provides two types of realms, namely internal realms and external realms.

The two types of built-in internal realms are native and file. The native realm is the default realm, and the user credentials are stored in a special index called .security-7 on Elasticsearch itself. These users are managed using the User Management API or the Management page of the Kibana UI. We will be exploring this in more detail later in this chapter.

If the realm is of the file type, then the user credentials are stored in a file on each node. These users are managed via dedicated tools that are provided by X-Pack. These tools can be found at $ES_HOMEin. The files are stored under the $ES_HOMEconfig folder. Since the credentials are stored in a file, it is the responsibility of the administrator to create users with the same credentials on each node.

The built-in external realms are ldap, active_directory, and pki, which use the external LDAP server, the external Active Directory Server, and the Public Key Infrastructure, respectively, to authenticate users. 

Depending on the realms that have been configured, the user credentials need to be attached to the requests that are sent to Elasticsearch. Realms live within a realm chain. The realm's order is configured in the elasticsearch.yml file and determines the order in which realms are consulted during the authentication process. Each realm is consulted one by one based on the order defined until the authentication is successful. Once one of the realms successfully authenticates the request, the authentication is considered to be successful. If none of the realms are able to authenticate the user, then the authentication is considered unsuccessful and an authentication error (HTTP 401) will be returned to the caller. The default realm chain consists of internal realm types, that is, native and file.

If none of these realms are specified in elasticsearch.yml, then the default realm that's used is native. To use the file type realm or external realms, they need to be specified in the elasticsearch.yml file.

For example, the following snippet shows the configuration for the realm chain containing native, file, and ldap:

xpack.security.authc:
  realms:
    native:
      type: native
      order: 0
    
    file:
      type: file
      order: 1
    
    ldap_server:
      type: ldap
      order: 2
      url: 'url_to_ldap_server'