OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add elastic/agent-skills --skill elasticsearch-security-troubleshooting
Diagnose and resolve Elasticsearch security errors: 401/403 failures, TLS problems, expired API keys, role mapping mismatches, and Kibana login issues. Use when the user reports a security error.
npx skills add elastic/agent-skills --skill elasticsearch-security-troubleshooting
Diagnose and resolve common Elasticsearch security issues. This skill provides a structured triage workflow for authentication failures, authorization errors, TLS problems, API key issues, role mapping mismatches, Kibana login failures, and license-expiry lockouts.
For authentication methods and API key management, see the elasticsearch-authn skill. For roles, users, and role mappings, see the elasticsearch-authz skill. For license management, see the elasticsearch-license skill.
For diagnostic API endpoints, see references/api-reference.md [blocked].
Deployment note: Diagnostic API availability differs between self-managed, ECH, and Serverless. See Deployment Compatibility for details.
| Item | Description |
|---|---|
| Elasticsearch URL | Cluster endpoint (e.g. https://localhost:9200 or a Cloud deployment URL) |
| Authentication | Any valid credentials — even minimal — to reach the cluster |
| Cluster privileges | monitor for read-only diagnostics; manage_security for fixes |
Prompt the user for any missing values. If the user cannot authenticate at all, start with TLS and Certificate Errors or License Expiry Recovery.
Route the symptom to the correct section:
| Symptom | Section |
|---|---|
HTTP 401, authentication_exception | Authentication Failures |
HTTP 403, security_exception, access denied | Authorization Failures |
| SSL/TLS handshake error, certificate rejected | TLS and Certificate Errors |
| API key rejected, expired, or ineffective | API Key Issues |
| Role mapping not granting expected roles | Role Mapping Issues |
| Kibana login broken, redirect loop, CORS error | Kibana Authentication Issues |
| All users locked out, paid features disabled | License Expiry Recovery |
Each section follows a Gather - Diagnose - Resolve pattern.
Use these APIs at the start of any security investigation:
Confirms identity, realm, and roles. If this fails with 401, the problem is authentication.
Confirms whether security is enabled (features.security.enabled). If security is disabled, all security APIs return
errors.
Tests whether the authenticated user holds specific privileges without requiring manage_security.
Check license type and status. An expired paid license disables paid realms and features.
A 401 response means Elasticsearch could not verify the caller's identity.
The -v flag shows headers and the response body. Look for:
WWW-Authenticate header — indicates which auth schemes the cluster accepts.authentication_exception in the response body — the reason field describes what failed.| Symptom | Likely cause |
|---|---|
unable to authenticate user | Wrong username or password |
unable to authenticate with provided credentials | Credentials do not match any realm in the chain |
user is not enabled | The native user account is disabled |
token is expired | API key or bearer token has expired |
No WWW-Authenticate header | Security may be disabled; check GET /_xpack |
If the user authenticates via an external realm (LDAP, AD, SAML, OIDC), the realm chain order matters. Elasticsearch tries realms in configured order and stops at the first match. If a higher-priority realm rejects the credentials before the intended realm is reached, authentication fails.
| Cause | Action |
|---|---|
| Wrong credentials | Verify username/password or API key value. See elasticsearch-authn. |
| Disabled user | PUT /_security/user/{name}/_enable. See elasticsearch-authz. |
| Expired API key | Create a new API key. See API Key Issues. |
| Realm chain order | Check elasticsearch.yml realm order (self-managed only). |
| Security disabled | Enable xpack.security.enabled: true in elasticsearch.yml and restart. |
| Paid realm after expiry | License expired — see License Expiry Recovery. |
A 403 response means the user is authenticated but lacks the required privileges.
Test the specific privileges the operation requires:
The response contains a has_all_requested boolean and per-resource breakdowns.
Also check the user's effective roles:
Inspect the roles array and authentication_realm to confirm the user is who you expect.
| Symptom | Likely cause |
|---|---|
has_all_requested: false for an index | Role is missing the required index privilege |
has_all_requested: false for a cluster | Role is missing the required cluster privilege |
| User has fewer roles than expected | Roles array was replaced (not merged) on last update |
| API key returns 403 on previously allowed | API key privileges are a snapshot — role changes after |
| operation | creation do not propagate to existing keys |
| Cause | Action |
|---|---|
| Missing index privilege | Add the privilege to the role or create a new role. See elasticsearch-authz. |
| Missing cluster privilege | Add the cluster privilege. See elasticsearch-authz. |
| Roles replaced on update | Fetch current roles first, then update with the full array. See elasticsearch-authz. |
| Stale API key privileges | Create a new API key with updated role_descriptors. See elasticsearch-authn. |
TLS errors prevent the client from establishing a connection at all.
Look for:
SSL certificate problem: unable to get local issuer certificate — CA not trusted.SSL certificate problem: certificate has expired — certificate past its validity date.SSL: no alternative certificate subject name matches target host name — hostname mismatch.For deeper inspection (self-managed only):
This displays the full certificate chain, expiry dates, and subject alternative names.
| Error message | Likely cause |
|---|---|
unable to get local issuer certificate | Missing or wrong CA certificate |
certificate has expired | Server or CA certificate past expiry |
no alternative certificate subject name matches | Certificate SAN does not include the hostname |
self-signed certificate | Self-signed cert not in the trust store |
SSLHandshakeException (Java client) | Truststore missing the CA or wrong password |
| Cause | Action |
|---|---|
| Wrong CA cert | Pass the correct CA with --cacert or add it to the system trust store. |
| Expired certificate | Regenerate certificates with elasticsearch-certutil (self-managed). |
| Hostname mismatch | Regenerate the certificate with the correct SAN entries. |
| Self-signed cert | Distribute the CA cert to all clients or use a publicly trusted CA. |
| Quick workaround | Use curl -k / --insecure to skip verification. Not for production. |
On ECH, TLS is managed by Elastic — certificate errors usually indicate the client is not using the correct Cloud endpoint URL. On Serverless, TLS is fully managed and transparent.
Retrieve the key's metadata:
Check expiration, invalidated, and role_descriptors in the response.
| Symptom | Likely cause |
|---|---|
| 401 when using the key | Key expired or invalidated |
| 403 on operations that should be allowed | Key was created with insufficient role_descriptors |
| Derived key has no access | API key created another API key — derived keys have no privilege |
| Key works for some indices but not others | role_descriptors scope is too narrow |
| Cause | Action |
|---|---|
| Expired key | Create a new key with appropriate expiration. See elasticsearch-authn. |
| Invalidated key | Create a new key. Invalidated keys cannot be reinstated. |
| Wrong scope | Create a new key with correct role_descriptors. See elasticsearch-authn. |
| Derived key problem | Use POST /_security/api_key/grant with user credentials instead. See elasticsearch-authn. |
Role mappings grant roles to users from external realms. When they fail silently, users authenticate but get no roles.
Note the username, authentication_realm.name, and roles array.
List all mappings and inspect their rules and enabled fields.
| Symptom | Likely cause |
|---|---|
User has empty roles array | No mapping matches the user's attributes |
| User gets wrong roles | A different mapping matched first or the rule is too broad |
| Mapping exists but does not apply | enabled is false |
| Mustache template produces wrong role name | Template syntax error or unexpected attribute value |
Compare the user's authentication_realm.name and groups (from _authenticate) against each mapping's rules to
find the mismatch.
| Cause | Action |
|---|---|
| No matching rule | Update the mapping rules to match the user's realm and attributes. |
| Mapping disabled | Set "enabled": true on the mapping. |
| Template error | Test the Mustache template with known attribute values. See elasticsearch-authz. |
| Rule too broad | Add all / except conditions to narrow the match. See elasticsearch-authz. |
kbn-xsrf headerAll mutating Kibana API requests require the kbn-xsrf header:
Without it, Kibana returns 400 Bad Request with "Request must contain a kbn-xsrf header".
Common causes:
xpack.security.authc.realms.saml.*.sp.acs or idp.metadata.path in elasticsearch.yml.server.publicBaseUrl does not match the SAML ACS URL.Verify the SAML realm configuration:
If this returns a valid user via a non-SAML realm, the SAML realm itself is not being reached. Check realm chain order.
Kibana logs Unable to retrieve version information from Elasticsearch nodes. Verify the elasticsearch.hosts setting
in kibana.yml points to a reachable endpoint and the credentials (elasticsearch.username / elasticsearch.password
or elasticsearch.serviceAccountToken) are valid.
When a paid license expires, the cluster enters a security-closed state: paid realms (SAML, LDAP, AD, PKI) stop working and users authenticating through them are locked out. Native and file realms remain functional.
If license.status is "expired", proceed with recovery.
Follow the detailed recovery workflow in the elasticsearch-license skill. The critical first step depends on deployment type:
| Deployment | First step |
|---|---|
| Self-managed | Log in with a file-based user (elasticsearch-users CLI) or native user. |
| ECH | Contact Elastic support or renew via the Cloud console. |
| Serverless | Not applicable — licensing is fully managed by Elastic. |
Symptom: "I get a 403 when searching logs-*."
Response shows "roles": ["viewer"].
Response: "has_all_requested": false — the viewer role does not include read on logs-*.
logs-reader role and assign it to Joe. See elasticsearch-authz.Symptom: "My API key returns 401 since yesterday."
Response shows "expiration": 1709251200000 — the key expired.
expiration. See elasticsearch-authn.Symptom: "Clicking the SSO button in Kibana redirects to an error page."
server.publicBaseUrl in kibana.yml matches the SAML ACS URL configured in the IdP.Symptom: "Nobody can log in to Kibana. We use SAML."
Response shows "status": "expired", "type": "platinum".
_authenticateRun GET /_security/_authenticate as the first diagnostic step. It reveals the user's identity, realm, roles, and
authentication type in a single call. Most issues become apparent from this response alone.
Before investigating realm or privilege issues, verify the license is active with GET /_license. An expired paid
license disables realms and features, producing symptoms that mimic misconfiguration.
_has_privileges before manual inspectionInstead of reading role definitions and mentally computing effective access, use POST /_security/user/_has_privileges
to test specific privileges directly. This is faster and accounts for role composition, DLS, and FLS.
Never use the built-in elastic superuser for day-to-day troubleshooting. Create a dedicated admin user or API key with
manage_security privileges. Reserve the elastic user for initial setup and emergency recovery only.
Using curl -k or --insecure skips certificate verification and masks real TLS issues. Use it only for initial
diagnosis, then fix the underlying certificate problem.
Diagnostic tool and API availability differs across deployment types.
| Tool / API | Self-managed | ECH | Serverless |
|---|---|---|---|
_security/_authenticate | Yes | Yes | Yes |
_security/user/_has_privileges | Yes | Yes | Yes |
_xpack | Yes | Yes | Limited |
_license | Yes | Yes (read) | Not available |
_security/api_key (GET) | Yes | Yes | Yes |
_security/role_mapping | Yes | Yes | Yes |
elasticsearch-users CLI | Yes | Not available | Not available |
openssl s_client on nodes | Yes | Not available | Not available |
| Elasticsearch logs | Yes | Via Cloud UI | Via Cloud UI |
ECH notes:
elasticsearch-users CLI and direct log/certificate inspection are not available.Serverless notes:
curl <auth_flags> "${ELASTICSEARCH_URL}/_license"curl <auth_flags> "${ELASTICSEARCH_URL}/_license"curl -v <auth_flags> "${ELASTICSEARCH_URL}/_security/_authenticate" 2>&1curl -X POST "${ELASTICSEARCH_URL}/_security/user/_has_privileges" \
<auth_flags> \
-H "Content-Type: application/json" \
-d '{
"index": [
{ "names": ["logs-*"], "privileges": ["read", "view_index_metadata"] }
],
"cluster": ["monitor"]
}'curl -v --cacert "${CA_CERT}" "https://${ELASTICSEARCH_HOST}:9200/" 2>&1 | head -30openssl s_client -connect "${ELASTICSEARCH_HOST}:9200" -showcerts </dev/null 2>&1curl "${ELASTICSEARCH_URL}/_security/api_key?name=${KEY_NAME}" <auth_flags>curl <auth_flags> "${ELASTICSEARCH_URL}/_security/role_mapping"curl -X PUT "${KIBANA_URL}/api/security/role/my-role" \
<auth_flags> \
-H "kbn-xsrf: true" \
-H "Content-Type: application/json" \
-d '{ ... }'curl -u "joe:${PASSWORD}" "${ELASTICSEARCH_URL}/_security/_authenticate"curl -X POST "${ELASTICSEARCH_URL}/_security/user/_has_privileges" \
-u "joe:${PASSWORD}" \
-H "Content-Type: application/json" \
-d '{"index": [{"names": ["logs-*"], "privileges": ["read"]}]}'curl -u "admin:${PASSWORD}" "${ELASTICSEARCH_URL}/_security/api_key?name=my-key"curl -u "elastic:${PASSWORD}" "${ELASTICSEARCH_URL}/_security/_authenticate"curl -s "${IDP_METADATA_URL}" | head -5curl -u "admin:${PASSWORD}" "${ELASTICSEARCH_URL}/_license"