# index.html.md # Charmed OpenSearch documentation Charmed OpenSearch is an open-source operator, packaged as a [Juju charm](https://documentation.ubuntu.com/juju/3.6/reference/charm/), that simplifies the deployment, scaling, and management of [OpenSearch](http://opensearch.org/) clusters on physical hardware, VMs, and cloud environments including AWS, Azure, OpenStack, and VMware. The charm automates OpenSearch operations from [Day 0 to Day 2](https://codilime.com/blog/day-0-day-1-day-2-the-software-lifecycle-in-the-cloud-age/) with capabilities such as TLS encryption, user management, horizontal scaling, replication, password rotation, monitoring, and application integration. ## In this documentation Charmed OpenSearch documentation has the following topics. ### Get started | | | |------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Getting started** | [Requirements](https://canonical.com/data/opensearch/docs/2//reference/system-requirements.md#reference-system-requirements) • [Tutorial: Introduction](https://canonical.com/data/opensearch/docs/2//tutorial/index.md#tutorial-index) • [Step 1: Environment setup](https://canonical.com/data/opensearch/docs/2//tutorial/1-set-up-the-environment.md#tutorial-1-set-up-the-environment) | | **Deployment** | [Standard deployment](https://canonical.com/data/opensearch/docs/2//how-to/deploy/standard.md#how-to-deploy-standard) • [Large deployment](https://canonical.com/data/opensearch/docs/2//how-to/deploy/large.md#how-to-deploy-large) | | **Cluster management** | [Horizontal scaling](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally) • [Applications integration](https://canonical.com/data/opensearch/docs/2//how-to/integrate-with-an-application.md#how-to-integrate-with-an-application) • [Version upgrades](https://canonical.com/data/opensearch/docs/2//how-to/upgrade.md#how-to-minor-upgrade) • [Version rollback](https://canonical.com/data/opensearch/docs/2//how-to/upgrade.md#how-to-minor-rollback) | | **Monitoring & performance** | [Monitoring](https://canonical.com/data/opensearch/docs/2//how-to/monitoring.md#how-to-monitoring) • [SMTP notifications](https://canonical.com/data/opensearch/docs/2//how-to/add-smtp-credentials.md#how-to-guides-add-smtp-credentials) • [Performance optimization](https://canonical.com/data/opensearch/docs/2//how-to/optimize-cluster-performance.md#how-to-optimize-cluster-performance) • [Load testing](https://canonical.com/data/opensearch/docs/2//how-to/perform-load-testing.md#how-to-perform-load-testing) • [OpenSearch Dashboards](https://canonical.com/data/opensearch/docs/2//dashboards/docs/index.md#dashboard-index) | | **Backups** | [Create a backup](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-create-a-backup) • [Azure configuration](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-azure-storage.md#how-to-back-up-configure-azure-storage) • [S3 configuration](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-s3.md#how-to-back-up-configure-s3) • [GCS configuration](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-gcs-storage.md#how-to-back-up-configure-gcs-storage) • [Restore from a local backup](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-restore-a-local-backup) • [Migrate a cluster](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-migrate-a-cluster) • [Recover from attached storage](https://canonical.com/data/opensearch/docs/2//how-to/persistent-storage.md#how-to-persistent-storage) | | **Security** | [Overview](https://canonical.com/data/opensearch/docs/2//explanation/security/index.md#explanation-security-index) • [Enable encryption](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-enable-tls-encryption) • [Rotate certificates](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-rotate-tls-ca-certificates) • [Manage passwords](https://canonical.com/data/opensearch/docs/2//how-to/manage-passwords.md#how-to-manage-passwords) • [OAuth](https://canonical.com/data/opensearch/docs/2//how-to/access-using-oauth.md#how-to-access-using-oauth) • [JWT Auth](https://canonical.com/data/opensearch/docs/2//how-to/enable-jwt-authentication.md#how-to-guides-enable-jwt-authentication) • [LDAP](https://canonical.com/data/opensearch/docs/2//how-to/enable-ldap-authentication.md#how-to-enable-ldap-authentication) • [Cryptography](https://canonical.com/data/opensearch/docs/2//explanation/security/cryptography.md#explanation-security-cryptography) | | **Internals** | [Node roles](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles) • [Cluster health](https://canonical.com/data/opensearch/docs/2//explanation/cluster-health.md#explanation-cluster-health) • [Performance profiles](https://canonical.com/data/opensearch/docs/2//explanation/performance-profiles.md#explanation-performance-profiles) • [Persistent storage](https://canonical.com/data/opensearch/docs/2//explanation/persistent-storage.md#explanation-persistent-storage) • [TLS certificates](https://canonical.com/data/opensearch/docs/2//explanation/security/tls-certificates.md#explanation-tls-certificates) • [Alert rules](https://canonical.com/data/opensearch/docs/2//reference/alert-rules.md#ref-alert-rules) • [Monitoring overview](https://canonical.com/data/opensearch/docs/2//explanation/monitoring.md#explanation-monitoring) • [Software testing](https://canonical.com/data/opensearch/docs/2//explanation/software-testing.md#explanation-software-testing) • [Release notes](https://canonical.com/data/opensearch/docs/2//reference/release-notes/index.md#reference-release-notes-index) | | **Contributing** | [Contribute](https://canonical.com/data/opensearch/docs/2//contributing.md#contributing-guide) | ## How the documentation is organized This documentation uses the [Diátaxis documentation structure](https://diataxis.fr/): - The [Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/index.md#tutorial-index) walks you through deploying your first Charmed OpenSearch cluster from scratch, step by step. - [How-to guides](https://canonical.com/data/opensearch/docs/2//how-to/index.md#how-to-index) help you solve specific operational tasks such as enabling TLS, scaling, or integrating with other applications. - [Reference](https://canonical.com/data/opensearch/docs/2//reference/index.md#reference-index) lets you look up system requirements, release notes, and configuration options. - [Explanation](https://canonical.com/data/opensearch/docs/2//explanation/index.md#explanation-index) helps you understand the design decisions behind security, monitoring, and software testing. ## Project & community Charmed OpenSearch is an official distribution of OpenSearch. It’s an open-source project that welcomes community contributions, suggestions, fixes and constructive feedback. ### Get involved - [Join the Discourse forum](https://discourse.charmhub.io/tag/opensearch) - [Chat with us on Matrix](https://matrix.to/#/#charmhub-data-platform:ubuntu.com) - [Report an issue](https://github.com/canonical/opensearch-operator/issues/new) - [Contribute](https://canonical.com/data/opensearch/docs/2//contributing.md#contributing-guide) to the code and documentation - Explore [Canonical’s open-source data platform](https://canonical.com/data) ### Governance and policies - [Read our Code of Conduct](https://ubuntu.com/community/code-of-conduct) - [Report a security issue](https://wiki.ubuntu.com/DebuggingSecurity#How_to_File) — please do not use GitHub issues for security topics - [Canonical Contributor Agreement](https://ubuntu.com/legal/contributors) ## License & trademark The Charmed OpenSearch ROCK, Charmed OpenSearch snap, and Charmed OpenSearch Operator are free software, distributed under the [Apache Software License, version 2.0](https://github.com/canonical/charmed-opensearch-rock/blob/2-24.04/edge/licenses/LICENSE-rock). They install and operate OpenSearch, which is also licensed under the [Apache Software License, version 2.0](https://github.com/canonical/charmed-opensearch-rock/blob/2-24.04/edge/licenses/LICENSE-opensearch). OpenSearch is a registered trademark of Amazon Web Services. Other trademarks are property of their respective owners. Charmed OpenSearch is not sponsored, endorsed, or affiliated with Amazon Web Services. # index.html.md # How to contribute Charmed OpenSearch is an open-source project developed and supported by [Canonical](https://canonical.com/) that welcomes community contributions, suggestions, fixes, and constructive feedback. If you would like to contribute a larger change, please [get in touch](#contributing-contact) with us first so we can help you shape the contribution. ## Report an issue Report bugs and feature requests on [GitHub](https://github.com/canonical/opensearch-operator/issues/new). For documentation issues, use the **Give feedback** button at the top of the relevant page to open a pre-filled GitHub issue. #### NOTE Please do **not** use GitHub issues for security topics. See [the section below](#contributing-security). ### Report a security issue Security issues should be reported through [Launchpad](https://wiki.ubuntu.com/DebuggingSecurity#How_to_File), following the Ubuntu security disclosure process. Please do **not** file GitHub issues on security topics. See also [SECURITY.md](https://github.com/canonical/opensearch-operator/blob/2/edge/SECURITY.md) in the repository. ## Get in touch If you have questions after reading this documentation or would like to discuss Charmed OpenSearch, get in touch through one of the following channels: - Chat with the Data team directly on [Matrix](https://matrix.to/#/#charmhub-data-platform:ubuntu.com). - Ask questions and share feedback on the [Discourse forum](https://discourse.charmhub.io/t/charmed-opensearch-documentation/9729). - To talk to Canonical about your use case or commercial support, use the [business form](https://canonical.com/data/opensearch#get-in-touch). ## Contribute code If you would like to contribute, the following sections cover the building and testing for both source code and documentation. ### Requirements To build the charm locally, you will need to install [Charmcraft](https://snapcraft.io/charmcraft) (or [charmcraftcache](https://github.com/canonical/charmcraftcache)), as well as [`tox`](https://tox.wiki/en/stable/) and [Poetry](https://python-poetry.org/). The easiest way to install the last two is with [pipx](https://pipx.pypa.io/stable/): ```bash pipx install tox pipx install poetry pipx install charmcraftcache ``` To run the machine charm locally with Juju, it is recommended to use [LXD](https://linuxcontainers.org/lxd/introduction/) as your virtual machine manager. Instructions for running Juju on LXD can be found [here](https://documentation.ubuntu.com/juju/3.6/reference/cloud/list-of-supported-clouds/lxd/). This repository is a monorepo containing two charms: - `machine/` — the machine charm (`opensearch`), which installs and manages OpenSearch from the [OpenSearch snap](https://snapcraft.io/opensearch) on VMs and machine clusters. - `kubernetes/` — the Kubernetes charm (`opensearch-k8s`), which deploys and manages OpenSearch as a container workload on Kubernetes. Both of these charms are built on top of the shared [`opensearch-single-kernel-library`](https://github.com/canonical/opensearch-single-kernel-library). The related Charmed OpenSearch Dashboards charms (which live in the separate [`opensearch-dashboards-operator`](https://github.com/canonical/opensearch-dashboards-operator) repository) are built on their own shared library, [`opensearch-dashboards-single-kernel-library`](https://github.com/canonical/opensearch-dashboards-single-kernel-library). Each charm is a self-contained project: build commands must be run from inside the corresponding directory. ### Host and model prerequisites For the machine charm on LXD, OpenSearch has a set of [system requirements](https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/) to function correctly. Some of those settings must be set using `cloudinit-userdata` on the model, while others must be set on the host machine. For the Kubernetes charm, use a Kubernetes model and follow the [Kubernetes setup guidance](https://github.com/canonical/opensearch-operator/blob/2/edge/README.md#kubernetes-charm-opensearch-k8s) instead of the LXD steps below. On the host machine, create a sysctl configuration file and apply it: ```bash sudo tee /etc/sysctl.d/opensearch.conf < cloudinit-userdata.yaml cloudinit-userdata: | postruncmd: - echo 'vm.max_map_count=262144' >> /etc/sysctl.conf - echo 'vm.swappiness=0' >> /etc/sysctl.conf - echo 'fs.file-max=1048576' >> /etc/sysctl.conf - sysctl -p EOF ``` #### NOTE Keep each `postruncmd` entry as a **string**. Cloud-init runs string entries through a shell, so the `>>` redirection works. Entries written as a YAML list are passed straight to `execve(3)` with no shell, so `>>` would become a literal argument to `echo` instead of appending to the file. Then create a new model and set the previously generated file in it: ```bash # Create a model juju add-model dev # Enable DEBUG logging juju model-config logging-config="=INFO;unit=DEBUG" # Add cloudinit-userdata juju model-config --file=./cloudinit-userdata.yaml # Increase the frequency of the update-status event juju model-config update-status-hook-interval=1m ``` ### Build and deploy To build a charm, enter the corresponding directory and pack it: ```bash # Clone and enter the repository git clone https://github.com/canonical/opensearch-operator.git cd opensearch-operator/machine # or: cd opensearch-operator/kubernetes # Build the charm locally charmcraftcache pack ``` In a model for the chosen substrate, deploy the Ubuntu 24.04 artifact with a TLS relation (packing also produces a 22.04 artifact): ```bash # Deploy the self-signed-certificates operator juju deploy self-signed-certificates --channel=1/stable --show-log --verbose # Generate a CA certificate juju config self-signed-certificates ca-common-name="CN_CA" ``` From `machine/`, deploy and relate the machine charm: ```bash juju deploy -n 1 ./opensearch_ubuntu@24.04-amd64.charm --show-log --verbose juju integrate opensearch:certificates self-signed-certificates ``` Alternatively, from `kubernetes/` in a Kubernetes model, supply the workload image declared in `metadata.yaml` when deploying the locally packed charm: ```bash juju deploy -n 1 ./opensearch-k8s_ubuntu@24.04-amd64.charm \ --resource opensearch-image="$(awk '/upstream-source:/ {print $2}' metadata.yaml)" \ --show-log --verbose juju integrate opensearch-k8s:certificates self-signed-certificates ``` #### NOTE The TLS settings shown here are for self-signed-certificates, which are not recommended for production clusters. The TLS Certificates Operator offers a variety of configurations. Read more on the self-signed-certificates Operator [here](https://charmhub.io/self-signed-certificates). ### Develop and test Return to the repository root (`cd ..` after building a charm), then create a development environment and check your changes: ```bash poetry install tox run -e lint # check code style tox run -e format # apply formatting fixes, if needed ``` For charm-logic changes, clone the [`opensearch-single-kernel-library`](https://github.com/canonical/opensearch-single-kernel-library) and run `tox run -e unit` and `tox run -e integration` **there**, not in this repository. See [Software testing for charms](https://canonical.com/data/opensearch/docs/2//explanation/software-testing.md#explanation-software-testing) for details. This repository’s smoke-level integration tests run in [CI](https://github.com/canonical/opensearch-operator/blob/2/edge/.github/workflows/integration_test.yaml). The tutorial end-to-end test suite (requires [Multipass](https://documentation.ubuntu.com/multipass/) and [Spread](https://github.com/canonical/spread)) can be run from the repository root with: ```bash tox -e tutorial-extract # check tutorial commands without starting a VM tox -e tutorial # extract scripts + run the end-to-end tests in a VM ``` See [tests/tutorial/](https://github.com/canonical/opensearch-operator/tree/2/edge/tests/tutorial) for the extraction scripts and the generated tasks. #### NOTE The code blocks in the documentation tutorial pages are extracted and run as part of the tutorial test suite. When editing `docs/tutorial/*.md`, make sure `tox -e tutorial-extract` still succeeds. ### Review process All enhancements require review before being merged. Code review typically examines code quality, test coverage, and the user experience for Juju administrators of this charm. Please help us out in ensuring easy-to-review branches by rebasing your pull request branch onto the `2/edge` branch. This also avoids merge commits and creates a linear Git commit history. Familiarising yourself with the [Ops framework](https://canonical.com/juju/docs/ops/latest/) will help you when working on new features or bug fixes. ## Contribute documentation The documentation lives in the `docs/` folder of this repository and is built with [Sphinx](https://www.sphinx-doc.org/) from MyST Markdown sources. It is published on [canonical.com](https://canonical.com/data/opensearch/docs/). ### Prerequisites - A [GitHub account](https://docs.github.com/en/get-started/start-your-journey/creating-an-account-on-github). - Compliance with the [Code of Conduct](#contributing-code-of-conduct). ### Report a documentation issue To report an issue with spelling, grammar, or technical content, [file an issue on GitHub](https://github.com/canonical/opensearch-operator/issues/new) or use the **Give feedback** button at the top of the affected page. ### Make a contribution For a quick fix — a typo, a broken link, a small clarification — the easiest way is to click the pencil icon at the top of the documentation page (next to the **Give feedback** button). It takes you to the GitHub web editor for that page, where you can submit a pull request directly through the web interface. For larger contributions: 1. Create a branch (in the main repository or in a fork) from the current `2/edge` and modify the documentation files as necessary. 2. Raise a pull request against `2/edge` to start the review process. 3. Once the pull request is approved and all comments are addressed, it can be merged. To preview and test the documentation locally: ```bash cd docs make run # live-reload build served on http://127.0.0.1:8000 ``` Before submitting, make sure the following checks pass: ```bash cd docs make html # full build; fails on warnings make linkcheck # verify all external links make lint-md # Markdown linting make spelling # Vale spelling check make woke # inclusive-language check ``` #### NOTE The documentation for OpenSearch Dashboards lives in the [`opensearch-dashboards-operator`](https://github.com/canonical/opensearch-dashboards-operator) repository and is included here as a git submodule. Contribute Dashboards documentation changes upstream, in that repository. The documentation follows the [Diátaxis structure](https://diataxis.fr/): tutorials, how-to guides, reference, and explanation each live in their own section and should not be mixed. ## Code of conduct This project follows the [Ubuntu Code of Conduct](https://ubuntu.com/community/code-of-conduct). Maintainers reserve the right to remove any contributions that do not respect it. ## Contributor agreement Canonical welcomes contributions to Charmed OpenSearch. Please check out our [contributor agreement](https://ubuntu.com/legal/contributors) if you’re interested in contributing to the solution. ## We are hiring! Also, if you truly enjoy working on open-source projects like this one, check out the [career options](https://canonical.com/careers/all) we have at [Canonical](https://canonical.com/). ## Useful links - [Canonical Data solutions](https://canonical.com/data) - [Charmed OpenSearch](https://charmhub.io/opensearch) - [Charmed OpenSearch on Kubernetes](https://charmhub.io/opensearch-k8s) - [Git sources for Charmed OpenSearch](https://github.com/canonical/opensearch-operator) - [Canonical Data on Launchpad](https://launchpad.net/~data-platform) - [Canonical Data on Matrix](https://matrix.to/#/#charmhub-data-platform:ubuntu.com) # index.html.md # How to access OpenSearch using OAuth This guide shows how to authenticate OpenSearch requests with OAuth tokens issued by the Canonical Identity Platform (Hydra). ## Prerequisites - Minimum 4 CPUs, 16 GB RAM - Network access between the host and LXD containers - MicroK8s installed locally (for the Identity Platform) - A working LXD cloud with Juju bootstrapped (see the [Environment setup](https://canonical.com/data/opensearch/docs/2//tutorial/1-set-up-the-environment.md#tutorial-1-set-up-the-environment) stage of our tutorial for more details) ## Deploy OpenSearch on LXD Deploy OpenSearch and the `data-integrator` charm in a new model: ```shell juju add-model opensearch-model localhost/localhost juju deploy opensearch -n 3 --channel 2/stable juju deploy data-integrator --channel=stable \ --config index-name=admin-index \ --config extra-user-roles=admin ``` Wait until all units become active: ```shell juju status --watch 5s ``` ## Deploy the Identity Platform on MicroK8s The Identity Platform runs on Kubernetes. The steps below use MicroK8s, but any Juju-supported K8s cluster will work. ### Prepare MicroK8s Install MicroK8s and enable the required addons: ```shell sudo snap install microk8s --classic sudo microk8s enable hostpath-storage dns sudo microk8s enable metallb:10.0.0.2-10.0.0.5 ``` #### NOTE The MetalLB address range must provide at least two IP addresses (one for `traefik-public` and one for `traefik-admin`). If you are also running COS Lite or other LoadBalancer services on the same MicroK8s instance, increase the range accordingly. Add MicroK8s to your Juju controller: ```shell sudo microk8s config > microk8s-cluster.yaml export KUBECONFIG="$PWD/microk8s-cluster.yaml" juju add-k8s microk8s-cluster -c ``` ### Deploy Identity Platform Create a model on the MicroK8s cloud and deploy the Identity Platform bundle: ```shell juju add-model -c oauth microk8s-cluster/localhost juju deploy identity-platform --channel edge --trust true ``` Wait until all units become active. The `kratos-external-idp-integrator` will remain `blocked` — this is expected and does not affect OAuth functionality. ## Create cross-model integrations The Identity Platform and OpenSearch run in separate models (and potentially on different clouds). Use Juju cross-model offers to connect them. ### Offer certificates and OAuth from the Identity Platform model Switch to the Identity Platform model and offer the certificates and OAuth endpoints: ```shell juju switch oauth juju offer self-signed-certificates:certificates juju offer hydra:oauth ``` ### Consume and integrate in the OpenSearch model Switch to the OpenSearch model, consume the offers, and integrate them with OpenSearch: ```shell juju switch opensearch-model juju consume admin/oauth.self-signed-certificates juju consume admin/oauth.hydra juju integrate opensearch:certificates admin/oauth.self-signed-certificates juju integrate opensearch admin/oauth.hydra ``` ## Create an OAuth client and obtain a token With the Identity Platform running, create an OAuth client in Hydra and use it to request an access token. ### Create a client in Hydra Create an OAuth client: ```shell juju switch oauth juju run hydra/leader create-oauth-client \ grant-types='["client_credentials"]' \ audience='["opensearch"]' \ scope='["openid","profile","email","phone","offline"]' ``` Record the `client-id` and `client-secret` from the output. ### Get the Hydra public URL Retrieve the proxied endpoints from Traefik: ```shell juju run traefik-public/0 show-proxied-endpoints ``` Note the `hydra.url` value (e.g. `https://10.0.0.3/oauth-hydra`). Set environment variables: ```shell export OAUTH_CLIENT_ID= export OAUTH_CLIENT_SECRET= export HYDRA_URL= ``` ### Retrieve the CA certificate Both Hydra and OpenSearch serve TLS certificates issued by the `self-signed-certificates` charm, so the same CA verifies both. Save it to a file: ```shell juju run self-signed-certificates/leader get-ca-certificate --format=json \ | jq -r '.[].results."ca-certificate"' > cert.pem ``` #### NOTE Use `--cacert cert.pem` with every `curl` command below so that TLS certificates are verified. Do not use `curl -k` (`--insecure`), which disables verification entirely and exposes the request, including the client secret and the access token. ### Fetch an access token Request an access token from Hydra using the client credentials: ```shell curl --cacert cert.pem -u "${OAUTH_CLIENT_ID}:${OAUTH_CLIENT_SECRET}" \ -X POST "${HYDRA_URL}/oauth2/token" \ -d "scope=openid" \ -d "grant_type=client_credentials" \ -d "audience=opensearch" ``` Export the token from the JSON response: ```shell export OAUTH_ACCESS_TOKEN= ``` ### Test the token (before role mapping) Switch to the OpenSearch model and query the cluster with the token: ```shell juju switch opensearch-model export OPENSEARCH_ADDRESS="$(juju status | grep opensearch/0 | awk -F' ' '{print $5}')" curl --cacert cert.pem -H "Authorization: Bearer ${OAUTH_ACCESS_TOKEN}" \ "https://${OPENSEARCH_ADDRESS}:9200/_cat/indices" ``` Expect a `403 security_exception` — the client has no mapped roles yet. ```text {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:monitor/settings/get] and User [name=e9c3b483-90be-4843-b821-1152e40aaa0a, backend_roles=[], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [indices:monitor/settings/get] and User [name=e9c3b483-90be-4843-b821-1152e40aaa0a, backend_roles=[], requestedTenant=null]"},"status":403} ``` ### Get a username from the Data Integrator Retrieve credentials from the `data-integrator` charm: ```shell juju run data-integrator/0 get-credentials ``` Note the `username` field (e.g. `opensearch-client_4`): ```shell export DATA_INTEGRATOR_USER= ``` ## Configure role mapping Map the OAuth client ID to the Data Integrator user: ```shell juju config opensearch roles_mapping="{\"$OAUTH_CLIENT_ID\":\"$DATA_INTEGRATOR_USER\"}" ``` Wait for the charm to apply the change: ```shell juju status --watch 5s ``` ### Verify access Re-run the same request — it should now succeed: ```shell curl --cacert cert.pem -H "Authorization: Bearer ${OAUTH_ACCESS_TOKEN}" \ "https://${OPENSEARCH_ADDRESS}:9200/_cat/indices" ``` You should now receive a `200 OK` response with a list of indices, for example: ```text green open .plugins-ml-config QnsDThyaTAKw8cASRYeQMw 1 0 1 0 4kb 4kb green open .opensearch-observability coXcpdLWSOqbSQ136tbADg 1 0 0 0 208b 208b green open top_queries-2025.08.29-70656 GDHtcml_R6Okh2siIKgmPw 1 0 40 6 108kb 108kb green open .opendistro_security RPVY1SdfT_KzAPAX-aUCuw 1 0 10 1 71kb 71kb yellow open admin-index 1BQKqmjTQVa6_CeBTi53Gw 1 1 0 0 208b 208b green open .charm_node_lock 8KbPHHy3QneIW8uWbTuBhQ 1 0 1 0 4.1kb 4.1kb ``` ## Next steps - [Enable JWT authentication](https://canonical.com/data/opensearch/docs/2//how-to/enable-jwt-authentication.md#how-to-guides-enable-jwt-authentication) — an alternative token-based authentication method. - [Integrate with an application](https://canonical.com/data/opensearch/docs/2//how-to/integrate-with-an-application.md#how-to-integrate-with-an-application) — connect client applications using the `data-integrator` charm. # index.html.md # How-to guides These guides help you accomplish specific tasks with Charmed OpenSearch on machines. ## Deployment - [Standard deployment](https://canonical.com/data/opensearch/docs/2//how-to/deploy/standard.md#how-to-deploy-standard) - [Launch a large deployment](https://canonical.com/data/opensearch/docs/2//how-to/deploy/large.md#how-to-deploy-large) ## Security - [Manage TLS encryption](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-enable-tls-encryption) - [Manage passwords](https://canonical.com/data/opensearch/docs/2//how-to/manage-passwords.md#how-to-manage-passwords) - [Access OpenSearch using OAuth](https://canonical.com/data/opensearch/docs/2//how-to/access-using-oauth.md#how-to-access-using-oauth) - [Enable JWT authentication](https://canonical.com/data/opensearch/docs/2//how-to/enable-jwt-authentication.md#how-to-guides-enable-jwt-authentication) - [Enable LDAP authentication](https://canonical.com/data/opensearch/docs/2//how-to/enable-ldap-authentication.md#how-to-enable-ldap-authentication) ## Operations - [Scale a cluster horizontally](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally) - [Integrate with an application](https://canonical.com/data/opensearch/docs/2//how-to/integrate-with-an-application.md#how-to-integrate-with-an-application) - [Manage persistent storage](https://canonical.com/data/opensearch/docs/2//how-to/persistent-storage.md#how-to-persistent-storage) - [Optimize cluster performance](https://canonical.com/data/opensearch/docs/2//how-to/optimize-cluster-performance.md#how-to-optimize-cluster-performance) - [Enable email notifications](https://canonical.com/data/opensearch/docs/2//how-to/add-smtp-credentials.md#how-to-guides-add-smtp-credentials) ## Backup and restore - [Configure S3 storage](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-s3.md#how-to-back-up-configure-s3) - [Configure Azure storage](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-azure-storage.md#how-to-back-up-configure-azure-storage) - [Configure Google Cloud Storage](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-gcs-storage.md#how-to-back-up-configure-gcs-storage) - [Create and restore backups](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-create-a-backup) ## Upgrades - [Upgrade, rollback, and recover](https://canonical.com/data/opensearch/docs/2//how-to/upgrade.md#how-to-minor-upgrade) ## Monitoring - [Enable monitoring (COS)](https://canonical.com/data/opensearch/docs/2//how-to/monitoring.md#how-to-monitoring) - [Perform load testing](https://canonical.com/data/opensearch/docs/2//how-to/perform-load-testing.md#how-to-perform-load-testing) # index.html.md # How to enable JWT authentication This guide shows how to enable JSON Web Token (JWT) authentication for Charmed OpenSearch using the JWT integrator charm. To enable JWT authentication, you need to: 1. [Deploy and configure the JWT integrator](#jwt-deploy-and-configure). 2. [Integrate it with OpenSearch](#jwt-integrate-with-opensearch). ## Prerequisites - A running Charmed OpenSearch deployment (revision 275+ on 22.04, or 276+ on 24.04) - A valid JWT for testing, issued by your JWT provider - The signing key used to sign the JWT ## Deploy and configure the JWT integrator Deploy the charm: ```shell juju deploy jwt-integrator --channel 1/edge ``` The charm will be `blocked` until configured. Create a Juju secret with your signing key: ```shell juju add-secret jwt-key signing-key="" ``` Note the secret URI, then grant access and configure: ```shell juju grant-secret jwt-key jwt-integrator juju config jwt-integrator signing-key= ``` The `roles-key` option is **required** — the charm remains `blocked` until it is set. It specifies the JWT claim key from which OpenSearch extracts the user’s roles. Set it together with any additional options for your JWT provider (e.g. `subject-key`, `jwt-url-parameter`): ```shell juju config jwt-integrator roles-key= subject-key= jwt-url-parameter= ``` ## Integrate with OpenSearch to enable JWT authentication Connect the JWT integrator to OpenSearch: ```shell juju integrate jwt-integrator opensearch ``` After integration, both applications show `active` in `juju status`, and OpenSearch updates its security plugin. To verify, first save the cluster’s CA certificate chain to a file so that `curl` can verify the TLS certificate OpenSearch serves: ```shell juju run opensearch/leader get-password --format=json \ | jq -r '.[].results."ca-chain"' > cert.pem ``` Requests with a valid JWT bearer token now return `200 OK`: ```shell curl --cacert cert.pem -H "Authorization: Bearer " -XGET "https://:9200/_cat/nodes" ``` #### NOTE Do not use `curl -k` (`--insecure`) as a shortcut. It disables certificate verification and exposes the bearer token to man-in-the-middle interception. ## Large deployments In large deployments, integrate the JWT integrator with the **main orchestrator** application. Identify it from `juju status` integrations: ```text Integration provider Requirer Interface Type Message opensearch-main:peer-cluster-orchestrator opensearch-data:peer-cluster peer_cluster regular ``` Integrate: ```shell juju integrate jwt-integrator opensearch-main ``` If integrated with the wrong application, the charm shows `blocked` status. Remove the invalid relation and integrate with the main orchestrator. ## Use with OpenSearch Dashboards To enable JWT authentication in OpenSearch Dashboards: ```shell juju config jwt-integrator jwt-url-parameter=jwt juju integrate jwt-integrator opensearch-dashboards ``` Access the UI by appending the JWT as a URL parameter: ```text http://:5601?jwt= ``` ## Next steps - [Access OpenSearch using OAuth](https://canonical.com/data/opensearch/docs/2//how-to/access-using-oauth.md#how-to-access-using-oauth) — an alternative token-based authentication method. - [Security explanation](https://canonical.com/data/opensearch/docs/2//explanation/security/index.md#explanation-security-index) — background on authentication and TLS. # index.html.md # How to manage persistent storage This guide shows how to reuse disks that contain data from a previous OpenSearch cluster. For an explanation of the risks of disk reuse, dangling indices, and metadata cleanup, see [Persistent storage and disk recovery](https://canonical.com/data/opensearch/docs/2//explanation/persistent-storage.md#explanation-persistent-storage). #### NOTE Prefer snapshot and restore whenever possible. The procedures on this page for reusing a disk from a **different** cluster are last-resort disaster recovery, not a general-purpose migration path. To move data between clusters, always try [creating a backup](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-create-a-backup) and [restoring or migrating it](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-migrate-a-cluster) first. There are three scenarios: - **Same cluster** (routine, safe) — a detached volume is reattached to a new unit in the same cluster. The node rejoins automatically with no metadata changes required. - **Different cluster, attach** (last resort) — a disk from a different cluster is attached to an existing cluster. The disk holds stale metadata referencing the old cluster UUID. Only proceed if the source cluster is permanently gone and no viable snapshot exists — in that case, the last-resort `detach-cluster` tool discards that metadata so the node can join. - **Different cluster, bootstrap** (last resort) — a disk from a different cluster is used to seed a brand new single-node deployment. Only proceed under the same conditions as above — the last-resort `unsafe-bootstrap` tool resets the cluster metadata so that a new cluster UUID is assigned and the node can start. This guide applies only to **persistent** disks under Juju management (e.g. deployed using a persistent storage pool such as LXD ZFS or Btrfs). Non-persistent storage (such as the default `rootfs` pool) is destroyed when its unit is removed and cannot be reused. Bringing external disks or volumes into Juju is not currently supported. #### CAUTION Back up your data before proceeding. Reusing disks may cause older data to override newer data. ## Prerequisites Ensure the disks are visible within Juju. List volumes with: ```shell juju storage ``` Note the Storage ID of a detached volume for use in the steps below: ```text Unit Storage ID Type Pool Size Status Message opensearch-data/0 filesystem opensearch-pool 2.0 GiB detached opensearch/1 opensearch-data/1 filesystem opensearch-pool 2.0 GiB attached opensearch/2 opensearch-data/2 filesystem opensearch-pool 2.0 GiB attached ``` For more details, see [Juju storage management](https://juju.is/docs/juju/manage-storage). ## Reuse a disk within the same cluster Attach the detached volume to a new unit: ```shell juju add-unit opensearch --attach-storage opensearch-data/ ``` When the new unit shows `active/idle` in `juju status`, the node has rejoined the cluster with the existing data. Re-running `juju storage` shows the reused volume as `attached` to the new unit. To confirm from OpenSearch’s side, retrieve the admin password and CA certificate chain with `juju run opensearch/leader get-password`, saving the chain to a file (e.g. `cert.pem`), and list the cluster nodes: ```shell curl --cacert cert.pem -XGET "https://:9200/_cat/nodes" -u admin: ``` The new unit should appear in the output alongside the existing nodes. See [mapping Juju units to OpenSearch nodes](https://canonical.com/data/opensearch/docs/2//explanation/cluster-health.md#cluster-health-mapping-nodes) for how to read this output. ## Recover a disk from a different cluster (last resort) When attaching a disk from a different cluster, the node holds stale metadata referencing the old cluster UUID. The steps below perform coordination-metadata surgery on that disk and are a **last resort** — use them only when the source cluster is permanently gone and no viable snapshot exists to restore from instead. #### CAUTION `detach-cluster` and `unsafe-bootstrap` are last-resort disaster-recovery commands. OpenSearch warns that they can cause **arbitrary data loss**, because the node running the command may not hold the most recent cluster metadata. Only use them after the **permanent** loss of a majority (or all) of the `cluster_manager`-eligible nodes in a cluster, or after a brutal cluster decommission, and only when no viable snapshot recovery exists. A success message from either command does not mean no data was lost. Before proceeding, confirm all of the following: - The source cluster (or the majority of its `cluster_manager`-eligible nodes) is permanently lost, decommissioned, or otherwise unrecoverable — not merely offline or repairable by moving its data path to healthy hardware. - No usable snapshot exists to [restore or migrate](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-migrate-a-cluster) the data instead. - All other nodes that were part of the old cluster are stopped, if any survive. - If more than one node survives from the old cluster, `unsafe-bootstrap` should be run on the one reporting the highest `(term, version)` pair, since it holds the freshest metadata. Both `detach-cluster` and `unsafe-bootstrap` (used below) prompt for interactive confirmation (`Confirm [y/N]`) and print their own data-loss warning before making any change. ### Attach to an existing cluster Attach the used disk to a new unit: ```shell juju add-unit opensearch --attach-storage opensearch-data/ ``` The unit will fail to join. Connect to it to confirm this is the expected UUID-mismatch error and to run the remaining commands: ```shell juju ssh opensearch/ ``` All remaining commands in this section run **on the unit**, inside this session. Inspect the logs: ```shell sudo journalctl -u snap.opensearch.daemon -f ``` The unit repeatedly reports that it cannot join the cluster because of a UUID mismatch, similar to: ```text CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid ... rejecting ``` Press `Ctrl+C` to stop following the log, then stop the service: ```shell sudo systemctl stop snap.opensearch.daemon ``` Detach the node from its old cluster references: ```shell sudo -u snap_daemon \ OPENSEARCH_JAVA_HOME=/snap/opensearch/current/usr/lib/jvm/java-21-openjdk-amd64 \ OPENSEARCH_PATH_CONF=/var/snap/opensearch/current/etc/opensearch \ OPENSEARCH_HOME=/var/snap/opensearch/current/usr/share/opensearch \ OPENSEARCH_LIB=/var/snap/opensearch/current/usr/share/opensearch/lib \ OPENSEARCH_PATH_CERTS=/var/snap/opensearch/current/etc/opensearch/certificates \ /snap/opensearch/current/usr/share/opensearch/bin/opensearch-node detach-cluster ``` Restart the service: ```shell sudo systemctl start snap.opensearch.daemon ``` The node will join the cluster. To confirm, exit the unit session, retrieve the admin password and CA certificate chain with `juju run opensearch/leader get-password`, saving the chain to a file (e.g. `cert.pem`), and list the cluster nodes from the host with the [CAT nodes API](https://opensearch.org/docs/2.19/api-reference/cat/cat-nodes/): ```shell curl --cacert cert.pem -XGET "https://:9200/_cat/nodes" -u admin: ``` The recovered node should appear in the output alongside the existing nodes. See [mapping Juju units to OpenSearch nodes](https://canonical.com/data/opensearch/docs/2//explanation/cluster-health.md#cluster-health-mapping-nodes) for how to read this output. ### Bootstrap a new cluster from a used disk Deploy a new single-node cluster with the used disk: ```shell juju deploy opensearch --attach-storage opensearch-data/ ``` The unit will fail to start. Connect to it to confirm this is the expected error and to run the remaining commands: ```shell juju ssh opensearch/ ``` All remaining commands in this section run **on the unit**, inside this session. Inspect the logs: ```shell sudo journalctl -u snap.opensearch.daemon -f ``` The unit repeatedly reports that it is waiting for a cluster it cannot reach, similar to: ```text ConfigurationRepository: Wait for cluster to be available ``` Press `Ctrl+C` to stop following the log, then stop the service: ```shell sudo systemctl stop snap.opensearch.daemon ``` Run `unsafe-bootstrap` to reset cluster metadata: ```shell sudo -u snap_daemon \ OPENSEARCH_JAVA_HOME=/snap/opensearch/current/usr/lib/jvm/java-21-openjdk-amd64 \ OPENSEARCH_PATH_CONF=/var/snap/opensearch/current/etc/opensearch \ OPENSEARCH_HOME=/var/snap/opensearch/current/usr/share/opensearch \ OPENSEARCH_LIB=/var/snap/opensearch/current/usr/share/opensearch/lib \ OPENSEARCH_PATH_CERTS=/var/snap/opensearch/current/etc/opensearch/certificates \ /snap/opensearch/current/usr/share/opensearch/bin/opensearch-node unsafe-bootstrap ``` Restart the service: ```shell sudo systemctl start snap.opensearch.daemon ``` The cluster will form with a new UUID. To confirm, exit the unit session, retrieve the admin password and CA certificate chain with `juju run opensearch/leader get-password`, saving the chain to a file (e.g. `cert.pem`), and list the cluster nodes from the host with the [CAT nodes API](https://opensearch.org/docs/2.19/api-reference/cat/cat-nodes/): ```shell curl --cacert cert.pem -XGET "https://:9200/_cat/nodes" -u admin: ``` The single bootstrapped node should be listed as the elected cluster manager (marked `*`). See [mapping Juju units to OpenSearch nodes](https://canonical.com/data/opensearch/docs/2//explanation/cluster-health.md#cluster-health-mapping-nodes) for how to read this output. You can then add more units (fresh or detached from another cluster). ## Recover dangling indices After reattaching a used disk, check for indices that were not part of the current cluster using the [dangling indices API](https://opensearch.org/docs/2.19/api-reference/index-apis/dangling-index/). #### CAUTION The dangling indices API cannot guarantee that imported data represents the latest state of the data when the index was still part of the original cluster. #### CAUTION An `active/idle` status and a successful `detach-cluster` or `unsafe-bootstrap` message do **not** mean no data was lost. Always audit the recovered indices and document counts, and check for dangling indices, before treating the recovery as complete. ## Next steps - [Back up and restore](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-guides-back-up-and-restore-index) — the preferred, data-safe way to move data between clusters; create backups before reusing disks. - [Scale a cluster horizontally](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally) — safely remove units when reorganising storage. # index.html.md # How to enable monitoring (COS) This guide shows how to integrate Charmed OpenSearch with the Canonical Observability Stack (COS) for metrics, dashboards, alerts, and logs. The integration workflow depends on the charm variant: - On **VMs**, the `opensearch` machine charm exposes a single `cos-agent` endpoint, and a machine `grafana-agent` charm collects and forwards the telemetry to COS. - On **Kubernetes**, the `opensearch-k8s` charm exposes native COS endpoints (`metrics-endpoint`, `grafana-dashboard`, and `logging`), and a `grafana-agent-k8s` charm in the OpenSearch model collects and forwards the telemetry to COS. If COS Lite runs in the same model, OpenSearch can integrate with the COS applications directly instead. For background on monitoring features, see the [Monitoring explanation](https://canonical.com/data/opensearch/docs/2//explanation/monitoring.md#explanation-monitoring). ## Prerequisites ### VM * A deployed [Charmed OpenSearch cluster](https://canonical.com/data/opensearch/docs/2//tutorial/2-deploy-opensearch.md#tutorial-2-deploy-opensearch) * A deployed [`cos-lite` bundle in a Kubernetes environment](https://charmhub.io/topics/canonical-observability-stack/tutorials/install-microk8s), in a model separate from the OpenSearch model ### K8s * A deployed [`opensearch-k8s`](https://charmhub.io/opensearch-k8s) application with TLS configured * A deployed [`cos-lite` bundle](https://charmhub.io/topics/canonical-observability-stack/tutorials/install-microk8s), either in the same Kubernetes model as `opensearch-k8s` or in a separate model #### TIP The COS documentation recommends a dedicated model for COS Lite. If you deploy COS Lite in the OpenSearch model instead (`juju deploy cos-lite --trust`), skip the offer and consume steps below and [integrate the applications directly](#integrate-with-cos). ## Offer COS interfaces Switch to the COS K8s controller and offer the required interfaces. The easiest way is to deploy COS Lite with the [offers overlay](https://github.com/canonical/cos-lite-bundle/blob/main/overlays/offers-overlay.yaml), which creates cross-model offers named `grafana-dashboards`, `loki-logging`, and `prometheus-receive-remote-write`: ```shell juju switch : curl -L https://raw.githubusercontent.com/canonical/cos-lite-bundle/main/overlays/offers-overlay.yaml -O juju deploy cos-lite --trust --overlay ./offers-overlay.yaml ``` If COS Lite is already deployed without the overlay, offer the interfaces manually: ```shell juju offer grafana:grafana-dashboard grafana-dashboards juju offer loki:logging loki-logging juju offer prometheus:receive-remote-write prometheus-receive-remote-write ``` ## Consume offers from the OpenSearch model Switch to the OpenSearch model and consume the COS offers: ```shell juju switch : juju consume :admin/.grafana-dashboards juju consume :admin/.loki-logging juju consume :admin/.prometheus-receive-remote-write ``` ## Integrate with COS ### VM Deploy `grafana-agent` in the OpenSearch model: ```shell juju deploy grafana-agent ``` Integrate it with the consumed COS offers: ```shell juju integrate grafana-agent grafana-dashboards juju integrate grafana-agent loki-logging juju integrate grafana-agent prometheus-receive-remote-write ``` Integrate it with OpenSearch: ```shell juju integrate grafana-agent opensearch:cos-agent ``` ### K8s #### NOTE If `grafana-agent-k8s` is already deployed in this model (for example, while monitoring Charmed OpenSearch Dashboards), skip this step (agent deployment) and the next step (COS offers integration); only add the OpenSearch integrations below. Deploy [`grafana-agent-k8s`](https://charmhub.io/grafana-agent-k8s) in the OpenSearch model: ```shell juju deploy grafana-agent-k8s --trust ``` Integrate it with the consumed COS offers: ```shell juju integrate grafana-agent-k8s:grafana-dashboards-provider grafana-dashboards juju integrate grafana-agent-k8s:logging-consumer loki-logging juju integrate grafana-agent-k8s:send-remote-write prometheus-receive-remote-write ``` Integrate it with OpenSearch: ```shell juju integrate opensearch-k8s:metrics-endpoint grafana-agent-k8s:metrics-endpoint juju integrate opensearch-k8s:grafana-dashboard grafana-agent-k8s:grafana-dashboards-consumer juju integrate opensearch-k8s:logging grafana-agent-k8s:logging-provider ``` * `metrics-endpoint` lets the agent scrape the OpenSearch metrics endpoint. * `grafana-dashboard` transfers the **Charmed OpenSearch** dashboard. * `logging` sends the OpenSearch logs. If COS Lite is deployed in the **same model** as `opensearch-k8s`, skip the agent and the offers, and integrate OpenSearch with the COS applications directly: ```shell juju integrate opensearch-k8s:metrics-endpoint prometheus:metrics-endpoint juju integrate opensearch-k8s:grafana-dashboard grafana:grafana-dashboard juju integrate opensearch-k8s:logging loki:logging ``` After integration, Grafana will display the **Charmed OpenSearch** dashboard and Loki will receive OpenSearch logs. ### Large deployments ### VM For multi-application clusters, integrate `grafana-agent` with each OpenSearch application. The dashboard aggregates data from all connected units. ### K8s For multi-application clusters, repeat the three OpenSearch integrations (`metrics-endpoint`, `grafana-dashboard`, and `logging`) for each OpenSearch application. The dashboard aggregates data from all connected units. ### Multiple clusters Multiple deployments can share the same COS instance. The dashboard provides selectors to filter by cluster. ## Access the Grafana web interface Retrieve the Grafana admin password: ```shell juju run grafana/leader get-admin-password --model : ``` For detailed instructions, see [Browse dashboards](https://documentation.ubuntu.com/observability/track-3.0/tutorial/cos-lite-microk8s-sandbox/#browse-dashboards) in the COS tutorial. In Grafana, select the **Charmed OpenSearch** dashboard. You can filter by Juju model, application, unit, cluster, and node role. ![Charmed OpenSearch Grafana dashboard showing cluster health and shard counts](how-to/img/dash1.png) ![Charmed OpenSearch Grafana dashboard showing system usage and indexing metrics](how-to/img/dash2.png) To inspect OpenSearch logs, open **Explore** in Grafana, select the Loki data source, filter by `juju_application`, and run the query: ![Grafana Explore showing OpenSearch logs filtered by Juju application](how-to/img/cos-1.png) #### NOTE For exploring and visualising your indexed data (as opposed to cluster health metrics), deploy [Charmed OpenSearch Dashboards](https://canonical-charmed-opensearch-dashboards.readthedocs-hosted.com/). ## Next steps - [Perform load testing](https://canonical.com/data/opensearch/docs/2//how-to/perform-load-testing.md#how-to-perform-load-testing) — benchmark the cluster under load with COS monitoring. - [Monitoring explanation](https://canonical.com/data/opensearch/docs/2//explanation/monitoring.md#explanation-monitoring) — background on monitoring features. # index.html.md # How to integrate with an application This guide shows how to connect applications to Charmed OpenSearch, either through a Juju charm integration or via the `data-integrator` charm for non-Juju applications. ## Integrate a Juju charm with OpenSearch If you are developing a charm that needs to connect to OpenSearch, use the `opensearch_client` interface. ### Define the interface In your charm’s `metadata.yaml`: ```yaml requires: opensearch: interface: opensearch_client ``` ### Implement the interface in your charm Fetch the library: ```shell charmcraft fetch-lib charms.data_platform_libs.v0.data_interfaces ``` In your `charm.py`: ```python from charms.data_platform_libs.v0.data_interfaces import OpenSearchRequires class MyCharm(CharmBase): def __init__(self, *args): super().__init__(*args) self.opensearch = OpenSearchRequires(self, "opensearch", "my_index") self.framework.observe(self.opensearch.on.index_created, self._on_index_created) def _on_index_created(self, event): # Handle the index_created event pass ``` The `OpenSearchRequires` constructor accepts: - `charm` — the charm instance - `relation_name` — must match the name in `metadata.yaml` - `index` — the index name to connect to - `extra_user_roles` (optional) — additional roles for the user - `additional_secret_fields` (optional) — extra secret fields to share See the [`OpenSearchRequires` class](https://github.com/canonical/data-platform-libs/blob/main/lib/charms/data_platform_libs/v0/data_interfaces.py) in the `data_interfaces` library source for the full list of optional parameters. ### Create the integration Connect OpenSearch to your application: ```shell juju integrate opensearch ``` To remove: ```shell juju remove-relation opensearch ``` ## Integrate a non-Juju application with OpenSearch Use the [`data-integrator`](https://charmhub.io/data-integrator) charm to provide credentials and connection details to applications outside the Juju ecosystem. Deploy it: ```shell juju deploy data-integrator --config index-name= ``` Integrate with OpenSearch: ```shell juju integrate data-integrator opensearch ``` The charm will be `blocked` until the integration is established. To remove the integration: ```shell juju remove-relation data-integrator opensearch ``` ### Retrieve the credentials Once the integration is established, retrieve the connection credentials (username, password, endpoints, and CA certificate) by running the `get-credentials` action: ```shell juju run data-integrator/leader get-credentials ```
Output example ```yaml opensearch: endpoints: 10.95.38.139:9200,10.95.38.212:9200,10.95.38.94:9200 index: test-index password: j3JWFnDkoumCxn0CtKZRCmdRMUlYTZFI tls-ca: |- -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- username: opensearch-client_5 version: 2.14.0 ```
Use these credentials to connect your application to OpenSearch. For an example of connecting with `curl`, see the [Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/4-integrate-with-a-client-application.md#tutorial-4-integrate-with-a-client-application). ## Next steps - [Manage passwords](https://canonical.com/data/opensearch/docs/2//how-to/manage-passwords.md#how-to-manage-passwords) to rotate the credentials issued to the client. - [Enable monitoring (COS)](https://canonical.com/data/opensearch/docs/2//how-to/monitoring.md#how-to-monitoring) to observe the integrated workload. - [Manage TLS encryption](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-enable-tls-encryption) to secure client-to-node traffic. # index.html.md # How to scale a cluster horizontally This guide shows how to remove nodes from a Charmed OpenSearch cluster without data loss or availability disruption. For a hands-on example, see [Tutorial: Scale horizontally](https://canonical.com/data/opensearch/docs/2//tutorial/6-scale-horizontally.md#tutorial-6-scale-horizontally). #### WARNING * **Never remove multiple units in a single command.** Remove one unit at a time. * In highly available deployments, **do not scale below 3 nodes**. ## Scale up To add nodes to the cluster, run: ```shell juju add-unit opensearch -n ``` Monitor the new units joining the cluster: ```shell watch juju status ``` New units automatically join the cluster and start receiving shard allocations. No additional configuration is required. ## Scale down Before removing a node, verify the cluster is healthy. ### Via Juju The charm reflects cluster health in the application status on a best-effort basis: - `active` — the cluster is healthy - `maintenance` — shards are still initializing or relocating - `waiting` — the charm is waiting for an operation to complete - `blocked` — the cluster has issues (the message describes the problem) Run `juju status` and make sure the application is `active` before proceeding. A `maintenance` or `waiting` status does not necessarily mean something is wrong, but it does indicate that now is not the best time to scale down — wait for the cluster to settle. For example, the following output shows an application blocked because replica shards cannot be assigned: ```text App Version Status Scale Charm Channel Rev Exposed Message opensearch blocked 1 opensearch 2/stable 344 no 1 or more 'replica' shards are not assigned, please scale your application up. Unit Workload Agent Machine Public address Ports Message opensearch/0* active idle 1 10.95.38.230 9200/tcp ``` For an explanation of how each health state maps to a Juju status, see [Cluster health and scaling](https://canonical.com/data/opensearch/docs/2//explanation/cluster-health.md#explanation-cluster-health). The OpenSearch health API is the source of truth, so use the check below to confirm. ### Via the OpenSearch health API For more detail, query the [cluster health API](https://opensearch.org/docs/2.19/api-reference/cluster-api/cluster-health/). First, retrieve the admin credentials and the CA certificate: ```shell juju run opensearch/leader get-password ``` The `get-password` action returns the admin password and the CA certificate chain. Save the CA certificate to a file (e.g. `cert.pem`) to use with `curl`. ### Interpret cluster health The cluster health API returns `green`, `yellow`, or `red`. For a detailed explanation of what each state means and how it maps to Juju status, see [Cluster health and scaling](https://canonical.com/data/opensearch/docs/2//explanation/cluster-health.md#explanation-cluster-health). **`green`** — Scaling down is likely safe. Verify the target node does not hold a primary shard of an unreplicated index: ```shell curl --cacert cert.pem -XGET "https://:9200/_cat/shards" -u admin: ``` If it does, [re-route](https://opensearch.org/docs/latest/api-reference/cluster-api/cluster-reroute/) the shard to another node first. **`yellow`** — Scaling down **may not be safe**. Some replica shards are unassigned. Investigate with the [cluster allocation explain API](https://opensearch.org/docs/2.19/api-reference/cluster-api/cluster-allocation/): ```shell curl --cacert cert.pem -XGET "https://:9200/_cluster/allocation/explain" -u admin: ``` Depending on the cause, the resolution may be scaling up, adding storage to existing nodes, or [re-routing](https://opensearch.org/docs/latest/api-reference/cluster-api/cluster-reroute/) the affected shard to another node. The most common fix is to scale up to restore green health before scaling down: ```shell juju add-unit opensearch -n 1 ``` **`red`** — Scaling down **is not safe**. At least one primary shard is unassigned. Do not add units before you know why. Adding capacity only helps for some causes, and an empty node cannot recreate data that no longer exists. Diagnose the cause first: ```shell curl --cacert cert.pem -XGET "https://:9200/_cluster/allocation/explain" -u admin: ``` Then act on what the output reports: | Cause | Resolution | |-----------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Too little disk space, or too few suitable nodes | Scale up, or add storage to existing nodes. | | Allocation rules prevent the shard from being placed | Correct the allocation rules. | | `no_valid_shard_copy` — no valid copy of the primary shard exists on any available node | Recover the failed node, or [restore a snapshot](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-restore-a-local-backup). Adding units will not help. | Where the cause is capacity or topology, scale up: ```shell juju add-unit opensearch -n 1 ``` #### NOTE If health becomes `red` after removing a unit, the charm blocks further removal to give you the opportunity to scale back up. ### Remove one unit Once health is confirmed green, remove **a single unit**: ```shell juju remove-unit opensearch/ ``` Monitor progress: ```shell juju status --watch 1s ``` #### NOTE The charm does not know in advance whether a removal will degrade cluster health. Always verify health after each removal before proceeding. ### Verify and repeat After removal, OpenSearch reallocates the departed node’s shards across the remaining nodes. If the removed node was `cluster_manager`-eligible, the charm also updates the cluster’s voting configuration and a new cluster manager is elected. Wait for the application to fully stabilise (`active/idle`), then [check cluster health]() again. Repeat the removal step for each additional unit you need to remove. ## Next steps - [Optimize cluster performance with profiles](https://canonical.com/data/opensearch/docs/2//how-to/optimize-cluster-performance.md#how-to-optimize-cluster-performance) — tune resource allocation for the new cluster size. - [Upgrade, rollback, and recover](https://canonical.com/data/opensearch/docs/2//how-to/upgrade.md#how-to-minor-upgrade) — upgrade the cluster after scaling. # index.html.md # How to upgrade, rollback, and recover This guide shows how to perform a minor revision upgrade of a Charmed OpenSearch deployment, roll back if needed, and recover from a failed rollback. ## Perform a minor upgrade A minor upgrade is an upgrade from one minor version to another: OpenSearch `X.Y` -> OpenSearch `X.Y+1`. For example, from OpenSearch `2.15` to OpenSearch `2.16`. This guide will walk you through the steps to upgrade your OpenSearch cluster, including pre-upgrade checks, upgrading the OpenSearch cluster, preparing the application for the in-place upgrade, initiating the upgrade, resuming the upgrade, and checking the cluster’s health. #### CAUTION In large deployments, upgrades should follow a specific role-dependent order. **Upgrade all applications without the `cluster_manager` role first, then upgrade applications with the `cluster_manager` role.** The steps below describe upgrading a single application. In large deployments, repeat these steps for each application, following this order. ### Pre-upgrade checks Before upgrading your OpenSearch cluster, ensure that you have completed the following steps: 1. **Backup your data**: Before upgrading, back up your data to prevent data loss in case of failure. For more information, see [How to create a backup](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-create-a-backup). 2. **Make sure not to perform any extraordinary operations**: Avoid performing any concurrent operations on the cluster during the upgrade process. This can lead to an inconsistent state of the cluster. This includes: - Adding or removing units - Creating or destroying new relations - Changes in workload configuration - Upgrading other connected/related/integrated applications simultaneously - Backup / restore of snapshots ### Upgrade the OpenSearch cluster To upgrade your OpenSearch cluster, follow these steps: 1. Collect all necessary pre-upgrade information. It will be required for the rollback (if requested). **Do NOT skip this step**. 2. (optional) Scale-up: The new sacrificial unit will be the first to be updated, and will simplify the rollback procedure in the case of an upgrade failure. 3. Prepare the “Charmed OpenSearch” Juju application for the in-place upgrade. See the step description below for all the technical details the charm executes. 4. Upgrade: Only one app unit will be upgraded once started. In case of failure, roll back with `juju refresh`. 5. Resume upgrade: The upgrade can be resumed if the upgrade of the first unit is successful. All units in an app will be upgraded sequentially from the highest to lowest unit number. 6. (optional) Scale back: Remove no longer necessary units created in step 2 (if any). 7. Post-upgrade check: Ensure all units are in the proper state and the cluster is healthy. #### Collect all necessary pre-upgrade information The first step is to record the revision of the running application, as a safety measure for a rollback action. To accomplish this, run the `juju status` command and look for the deployed Charmed OpenSearch revision. The illustrative outputs below use `opensearch` as the application name on both substrates; the Kubernetes charm is `opensearch-k8s`. ### VM ```text Model Controller Cloud/Region Version SLA Timestamp dev localhost-localhost localhost/localhost 3.6.25 unsupported 10:16:46+01:00 App Version Status Scale Charm Channel Rev Exposed Message opensearch active 3 opensearch 2/stable 144 no self-signed-certificates active 1 self-signed-certificates latest/stable 155 no Unit Workload Agent Machine Public address Ports Message opensearch/0 active idle 0 10.214.176.180 9200/tcp opensearch/1 active idle 1 10.214.176.220 9200/tcp opensearch/2* active idle 2 10.214.176.175 9200/tcp self-signed-certificates/0* active idle 3 10.214.176.31 Machine State Address Inst id Base AZ Message 0 started 10.214.176.180 juju-0c35d2-0 ubuntu@24.04 Running 1 started 10.214.176.220 juju-0c35d2-1 ubuntu@24.04 Running 2 started 10.214.176.175 juju-0c35d2-2 ubuntu@24.04 Running 3 started 10.214.176.31 juju-0c35d2-3 ubuntu@24.04 Running ``` For this example, the current revision is **144** for OpenSearch. ### K8s ```text Model Controller Cloud/Region Version SLA Timestamp dev opensearch-k8s ck8s 3.6.28 unsupported 16:13:42+01:00 App Version Status Scale Charm Channel Rev Address Exposed Message opensearch active 3 opensearch-k8s 2/edge 8 10.152.183.239 no self-signed-certificates active 1 self-signed-certificates 1/stable 586 10.152.183.56 no Unit Workload Agent Address Ports Message opensearch/0 active idle 10.1.0.134 opensearch/1* active idle 10.1.0.203 opensearch/2 active idle 10.1.0.181 self-signed-certificates/0* active idle 10.1.0.84 ``` For this example, the current revision is **8** for OpenSearch. #### NOTE Make sure to store the revision number in case of rollback. If the deployment is of a local charm, save a copy of the current `.charm` file. #### Scale-up (optional) Optionally, it is recommended to scale the application up by one unit before upgrading. The new unit will be the first one to be updated, and it will assert that the upgrade is possible. In the event of a failure, an extra unit simplifies manual recovery without disrupting service. ### VM ```shell juju add-unit opensearch ``` ### K8s ```shell juju scale-application opensearch ``` Wait for the new unit to be up and ready. ### Prepare the application for the in-place upgrade 1. **IMPORTANT:** Create a backup of your cluster Refer to [How to create a backup](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-create-a-backup). 1. Perform the `pre-upgrade-check` action After the application has settled, it’s necessary to run the `pre-upgrade-check` action against the leader unit: ### VM ```shell juju run opensearch/leader pre-upgrade-check ``` The output should be similar to the following: ```text Running operation 1 with 1 task - task 2 on unit-opensearch-2 Waiting for task 2... result: Charm is ready for upgrade ``` ### K8s ```shell juju run opensearch/leader pre-upgrade-check ``` The output should be similar to the following: ```text Running operation 1 with 1 task - task 2 on unit-opensearch-1 Waiting for task 2... result: Charm is ready for upgrade ``` The action will ensure and check the health of OpenSearch and determine if the charm is well prepared to start an upgrade procedure. ### Initiate the upgrade Use the `juju refresh` command to trigger the charm upgrade process. You have control over what upgrade you want to apply: ### VM #### CAUTION Charmed OpenSearch supports performance profiles with different RAM consumption: * `production`: JVM heap set to 50% of the available RAM, capped at 31 GB * `testing`: JVM heap fixed at ~1 GB of RAM If the charm is running on a revision prior to `185`, the `testing` profile is the default. Ensure it is set before upgrading, then switch to a profile that suits your use case. See [How to optimize cluster performance with profiles](https://canonical.com/data/opensearch/docs/2//how-to/optimize-cluster-performance.md#how-to-optimize-cluster-performance). - You can upgrade the charm to the latest revision available in the charm store for a specific channel, in this case, the stable channel: ```shell # If your charm is running a revision prior to 185, then set the profile explicitly: juju refresh opensearch --channel 2/stable --config profile="testing" # Otherwise, just refresh juju refresh opensearch --channel 2/stable ``` - You can also upgrade the charm to a specific revision: ```shell juju refresh opensearch --revision 145 ``` - Or you can upgrade the charm using a local charm file: ```shell juju refresh opensearch --path /path/to/your/charm/file.charm ``` ### K8s - You can upgrade the charm to the latest revision available in the charm store for a specific channel, in this case, the edge channel: ```shell juju refresh opensearch --channel 2/edge ``` - You can also upgrade the charm to a specific revision: ```shell juju refresh opensearch --revision 14 ``` - Or you can upgrade the charm using a local charm file: ```shell juju refresh opensearch --path /path/to/your/charm/file.charm \ --resource opensearch-image= ``` The OpenSearch upgrade will execute only on the highest ordinal unit. For the running example, the illustrative `juju status` output will look similar to: ### VM ```text Model Controller Cloud/Region Version SLA Timestamp dev localhost-localhost localhost/localhost 3.6.25 unsupported 10:29:07+01:00 App Version Status Scale Charm Channel Rev Exposed Message opensearch blocked 4 opensearch 2/stable 145 no Upgrading. Verify highest unit is healthy & run `resume-upgrade` action. To rollback, `juju refresh` to last revision self-signed-certificates active 1 self-signed-certificates latest/stable 155 no Unit Workload Agent Machine Public address Ports Message opensearch/0 active idle 0 10.214.176.180 9200/tcp OpenSearch 2.15.0 running; Snap rev 56 (outdated); Charmed operator 1+e686854 opensearch/1 active idle 1 10.214.176.220 9200/tcp OpenSearch 2.15.0 running; Snap rev 56 (outdated); Charmed operator 1+e686854 opensearch/2* active idle 2 10.214.176.175 9200/tcp OpenSearch 2.15.0 running; Snap rev 56 (outdated); Charmed operator 1+e686854 opensearch/3 active idle 4 10.214.176.7 9200/tcp OpenSearch 2.16.0 running; Snap rev 57; Charmed operator 1+e686854 self-signed-certificates/0* active idle 3 10.214.176.31 ``` The highest unit (`opensearch/3`) is upgraded first. The application shows `blocked` with a message instructing you to verify the upgraded unit and run `resume-upgrade`. ### K8s ```text Model Controller Cloud/Region Version SLA Timestamp dev opensearch-k8s ck8s 3.6.28 unsupported 16:35:33+01:00 App Version Status Scale Charm Channel Rev Address Exposed Message opensearch blocked 4 opensearch-k8s 2/edge 14 10.152.183.239 no Upgrading. Verify highest unit is healthy & run `resume-upgrade` action. self-signed-certificates active 1 self-signed-certificates 1/stable 586 10.152.183.56 no Unit Workload Agent Address Ports Message opensearch/0 active idle 10.1.0.134 OpenSearch 2.19.5 running (restart pending); Charmed operator 1 opensearch/1* active idle 10.1.0.203 OpenSearch 2.19.5 running (restart pending); Charmed operator 1 opensearch/2 active idle 10.1.0.181 OpenSearch 2.19.5 running (restart pending); Charmed operator 1 opensearch/3 active idle 10.1.0.100 OpenSearch 2.19.6 running; Charmed operator 1 self-signed-certificates/0* active idle 10.1.0.84 ``` The highest unit (`opensearch/3`) is upgraded first. The application shows `blocked` with a message instructing you to verify the upgraded unit and run `resume-upgrade`. #### NOTE The unit should recover shortly after, but the time can vary depending on the amount of data written to the cluster while the unit was not part of the cluster. Be patient with large installations. ### Resume the upgrade After the first unit is upgraded, the charm will set the unit upgrade state as completed. If deemed necessary, you can further assert the success of the upgrade. If the unit is healthy within the cluster, the next step is to resume the upgrade process by running: ### VM ```shell juju run opensearch/leader resume-upgrade ``` ### K8s ```shell juju run opensearch/leader resume-upgrade ``` The `resume-upgrade` action will roll out the OpenSearch upgrade for the remaining units in the application. The action will be executed sequentially from the highest unit number to the lowest. Once all units are upgraded, the application status will return to `active`, all units will show `active`/`idle`, and the version messages will disappear. The `Rev` column in the `juju status` output will reflect the new charm revision. ### Rollback (optional) In case of a failed upgrade, you might potentially be able to rollback to the previous revision. To do so, follow the [Perform a minor rollback](#how-to-minor-rollback) section below. ### Scale-back (optional) If you scaled up the application in step 2, you can now scale it back down to the original number of units: ### VM ```shell juju remove-unit opensearch/ ``` ### K8s ```shell juju scale-application opensearch ``` ### Check the cluster health First, check the units have settled as `active`/`idle` in `juju status`, with the newer revision number in the `Rev` column. All unit messages should be empty (no version or upgrade messages). Check the cluster is healthy. OpenSearch’s upstream documentation [suggests the following check](https://opensearch.org/docs/2.19/install-and-configure/upgrade-opensearch/rolling-upgrade/). First, retrieve the admin credentials and the CA certificate chain: ### VM ```shell juju run opensearch/leader get-password ``` ### K8s ```shell juju run opensearch/leader get-password ``` Save the `ca-chain` value to a file (e.g. `cert.pem`) to use with `curl`: ```shell curl --cacert cert.pem -XGET "https://:9200/_cluster/health?pretty" -u admin: ``` The response should look similar to the following example: ```json { "cluster_name" : "opensearch-wvmy", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "discovered_master" : true, "discovered_cluster_manager" : true, "active_primary_shards" : 5, "active_shards" : 15, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ``` ## Perform a minor rollback #### CAUTION OpenSearch does not support downgrading. For more information, please refer to the upstream [OpenSearch documentation about rolling upgrades](https://docs.opensearch.org/latest/migrate-or-upgrade/rolling-upgrade/#preparing-to-upgrade). While rolling back a charm revision that does not change the underlying OpenSearch version is a safe operation, it is important to note that rolling back in Charmed OpenSearch is a best-effort process to restore the cluster to a previous revision. If the OpenSearch workload version is different, it does not guarantee that the cluster will be rolled back to a previous version. After a `juju refresh`, if there are any version incompatibilities in charm revisions, their dependencies, or any other unexpected failure in the upgrade process, the process will be halted and enter a failure state. Even if the underlying OpenSearch cluster continues to work, it’s important to roll back the charm to a previous revision so that an update can be attempted after further inspection of the failure. ### Pre-rollback checks To execute a rollback we take the same procedure as the upgrade, the difference being the charm revision to upgrade to. As an example follow up [the minor upgrades guide](#how-to-minor-upgrade). #### NOTE Do **not** run `pre-upgrade-check` before a rollback. The action refuses to run while an upgrade is in progress and fails with `Upgrade already in progress`. Because a rollback only happens mid-upgrade, the action can never succeed at this point. The charm runs the equivalent checks itself: after `juju refresh`, it detects the rollback and re-enables shard allocation without requiring the action. Before rolling back, check `juju status`. The application will show `blocked` with a message asking you to verify the highest unit and run `resume-upgrade`. Unit messages help identify which units have already been upgraded and which are still on the older workload. Note the current charm revision from the `Rev` column: **145** in the VM example above, or **14** in the Kubernetes example. The exact messages vary by charm and substrate. ### Rollback the charm #### CAUTION Do not trigger a rollback during a running upgrade action. It may cause an unpredictable OpenSearch state. #### CAUTION Rollbacks in Charmed OpenSearch are a best-effort process. It is recommended to perform a backup and restore to a new deployment with the desired OpenSearch version instead of performing a rollback. Rollbacks carry the potential of *data loss* and *downtime*. #### Rollback a charm revision with the same workload version Only use this procedure if the target revision runs the same OpenSearch version as the current revision on every unit. In the Kubernetes example above, the upgraded unit runs `2.19.6` and the other units run `2.19.5`, so that example belongs to the different-version rollback procedure below instead. For a same-version rollback, run `refresh` with the compatible previous charm revision. ### VM For example, to rollback to revision **144**, run: ```shell juju refresh opensearch --revision=144 ``` To deploy the previous revision’s `.charm` file: ```shell juju refresh opensearch --path= ``` After the refresh command, the Juju controller revision for the application will be back in sync with the running OpenSearch revision. `juju status` will show the application `active` with the previous revision number in the `Rev` column (e.g. **144**), and all units `active`/`idle` with no messages. ### K8s To roll back to a previous revision with the **same** OpenSearch workload version, run: ```shell juju refresh opensearch --revision= \ --resource opensearch-image= ``` To deploy the previous revision’s `.charm` file: ```shell juju refresh opensearch --path= \ --resource opensearch-image= ``` The highest unit is rolled back first. Once it is `active`, resume the rollback for the remaining units: ```shell juju run opensearch/leader resume-upgrade ``` After the resume-upgrade action, the Juju controller revision for the application will be back in sync with the running OpenSearch revision. `juju status` will show the application `active` with the previous revision number in the `Rev` column, and all units `active`/`idle` with no messages. #### Rollback a charm revision with a different workload version If you roll back to a charm revision with a different workload version, the process will roll back the charm code and then make a best-effort attempt to roll back the workload, since OpenSearch does not support downgrades. ##### If the rollback between the versions is possible In this case, both the charm code and the workload will be rolled back to the previous version. However, because rollback is a risky operation, rolling back the workload requires manual intervention. The charm will enter a `blocked` state and display a message instructing you to run the `force-refresh-start` action with `check-compatibility=false` to continue the best-effort workload rollback: ### VM ```text Model Controller Cloud/Region Version SLA Timestamp testing localhost-localhost localhost/localhost 3.6.25 unsupported 08:36:09+01:00 App Version Status Scale Charm Channel Rev Exposed Message opensearch blocked 3 opensearch 2/stable 344 no Upgrading. Verify highest unit is healthy & run `resume-upgrade` action. self-signed-certificates active 1 self-signed-certificates 1/stable 586 no Unit Workload Agent Machine Public address Ports Message opensearch/0 active idle 1 10.149.40.7 9200/tcp OpenSearch 2.19.4 running; Snap rev 98; Charmed operator 1+b55ac3966-dirty opensearch/1 active idle 2 10.149.40.93 9200/tcp OpenSearch 2.19.4 running; Snap rev 98; Charmed operator 1+b55ac3966-dirty opensearch/2* blocked idle 3 10.149.40.126 9200/tcp Rollback incompatible. Run 'juju run force-refresh-start' with `check-compatibility` set to false to override ... self-signed-certificates/0* active idle 0 10.149.40.252 ``` Units that had not yet upgraded their workload before the rollback (`opensearch/0` and `opensearch/1` above) simply run revision **344** normally. Only the unit that already advanced to the newer workload (`opensearch/2`) needs to roll that workload back and is blocked until you do. Run the action on the blocked unit: ```shell juju run opensearch/ force-refresh-start check-compatibility=false ``` ### K8s ```text Model Controller Cloud/Region Version SLA Timestamp dev opensearch-k8s ck8s 3.6.28 unsupported 10:22:45+01:00 App Version Status Scale Charm Channel Rev Address Exposed Message opensearch blocked 4 opensearch-k8s 2/edge 8 10.152.183.50 no Upgrading. Verify highest unit is healthy & run `resume-upgrade` action. self-signed-certificates active 1 self-signed-certificates 1/stable 586 10.152.183.237 no Unit Workload Agent Address Ports Message opensearch/0 active idle 10.1.0.232 OpenSearch 2.19.5 running (restart pending); Charmed operator 1 opensearch/1 active idle 10.1.0.242 OpenSearch 2.19.5 running (restart pending); Charmed operator 1 opensearch/2* active idle 10.1.0.239 OpenSearch 2.19.5 running (restart pending); Charmed operator 1 opensearch/3 blocked idle 10.1.0.73 Rollback incompatible. Run 'juju run force-refresh-start' with `check-compatibility` set to false to override ... self-signed-certificates/0* active idle 10.1.0.165 ``` Units that had not yet upgraded their workload before the rollback (`opensearch/0`, `opensearch/1` and `opensearch/2` above) simply run revision **8** normally. Only the unit that already advanced to the newer workload (`opensearch/3`) needs to roll that workload back and is blocked until you do. Run the action on the blocked unit: ```shell juju run opensearch/ force-refresh-start check-compatibility=false ``` Once it is `active`, run the `resume-upgrade` action: ```shell juju run opensearch/leader resume-upgrade ``` ##### If the rollback between the versions is not possible In this case, the charm code will be rolled back, but the OpenSearch workload will remain on the newer version. The charm will enter a `blocked` state and display a message instructing you to either refresh to a charm revision with the same workload version or perform a backup and restore to a new deployment: ### VM ```text Model Controller Cloud/Region Version SLA Timestamp testing localhost-localhost localhost/localhost 3.6.25 unsupported 08:03:52+01:00 App Version Status Scale Charm Channel Rev Exposed Message opensearch blocked 3 opensearch 2/stable 344 no Upgrading. Verify highest unit is healthy & run `resume-upgrade` action. self-signed-certificates active 1 self-signed-certificates 1/stable 586 no Unit Workload Agent Machine Public address Ports Message opensearch/0 active idle 1 10.149.40.7 9200/tcp OpenSearch 2.19.4 running; Snap rev 98; Charmed operator 1+b55ac3966-dirty opensearch/1 active idle 2 10.149.40.93 9200/tcp OpenSearch 2.19.4 running; Snap rev 98; Charmed operator 1+b55ac3966-dirty opensearch/2* blocked idle 3 10.149.40.126 9200/tcp Rollback unsupported. Refresh to a newer revision or consult the recovery documentation self-signed-certificates/0* active idle 0 10.149.40.252 ``` ### K8s ```text Model Controller Cloud/Region Version SLA Timestamp dev opensearch-k8s ck8s 3.6.28 unsupported 20:12:33+01:00 App Version Status Scale Charm Channel Rev Address Exposed Message opensearch blocked 4 opensearch-k8s 2/edge 8 10.152.183.109 no Upgrading. Verify highest unit is healthy & run `resume-upgrade` action. self-signed-certificates active 1 self-signed-certificates 1/stable 586 10.152.183.241 no Unit Workload Agent Address Ports Message opensearch/0* active idle 10.1.0.66 OpenSearch 2.19.5 running (restart pending); Charmed operator 1 opensearch/1 active idle 10.1.0.129 OpenSearch 2.19.5 running (restart pending); Charmed operator 1 opensearch/2 active idle 10.1.0.232 OpenSearch 2.19.5 running (restart pending); Charmed operator 1 opensearch/3 blocked idle 10.1.0.168 Rollback unsupported. Refresh to a newer revision or consult the recovery documentation self-signed-certificates/0* active idle 10.1.0.251 ``` ### Check the cluster’s health Once the charm is rolled back, it is important to check the cluster’s health to ensure it is healthy. OpenSearch’s upstream documentation [suggests the following check](https://opensearch.org/docs/2.19/install-and-configure/upgrade-opensearch/rolling-upgrade/): ```shell curl --cacert cert.pem -XGET "https://:9200/_cluster/health?pretty" -u admin: ``` The response should look similar to the following example: ```json { "cluster_name" : "opensearch-7ngj", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "discovered_master" : true, "discovered_cluster_manager" : true, "active_primary_shards" : 5, "active_shards" : 15, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ``` ## Recovering from a rollback #### WARNING This procedure applies to machine deployments. On Kubernetes, restore from backup to a new deployment as described in [How to back up and restore](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-migrate-a-cluster). OpenSearch does not support downgrades. Running `juju refresh` to a previous revision may cause OpenSearch to fail to start. In that case, manual recovery is required. Follow the steps in this section to restore the cluster to a healthy state. For more information, please refer to the upstream [OpenSearch documentation about rolling upgrades](https://docs.opensearch.org/latest/migrate-or-upgrade/rolling-upgrade/#preparing-to-upgrade). ### Check Juju status First, check Juju model status: ```shell juju status ``` The rolled back unit may appear stuck displaying the status `Waiting for OpenSearch to start...`: ```text Model Controller Cloud/Region Version SLA Timestamp test localhost-localhost localhost/localhost 3.6.25 unsupported 01:32:11+01:00 App Version Status Scale Charm Channel Rev Exposed Message opensearch active 3 opensearch 2/stable 168 no self-signed-certificates active 1 self-signed-certificates latest/stable 264 no Unit Workload Agent Machine Public address Ports Message opensearch/0* active idle 0 10.45.114.156 9200/tcp opensearch/1 active idle 1 10.45.114.208 9200/tcp opensearch/2 waiting executing 2 10.45.114.147 9200/tcp Waiting for OpenSearch to start... self-signed-certificates/0* active idle 3 10.45.114.124 ``` Note the blocked unit; in this example, it is `opensearch/2`. This unit will not recover automatically, and additional steps are required to replace it. ### Check cluster health Retrieve the cluster health using the `cert.pem` and `` obtained above: ```shell curl --cacert cert.pem -X GET "https://:9200/_cluster/health?pretty" -u admin: ``` If the cluster health is red, one or more primary shards cannot be allocated. Allocation explanations will identify any indices that exist only on the rolled back unit which has left the cluster. As the departed unit will not rejoin, these indices cannot be recovered and must be removed. Identify the problematic index from the output of: ```shell curl --cacert cert.pem -X GET "https://:9200/_cluster/allocation/explain?pretty" -u admin: ``` For example, in the following output, `index1` cannot be recovered as its current state is `unassigned` with the reason `NODE_LEFT`: ```json { "index": "index1", "shard": 0, "primary": true, "current_state": "unassigned", "unassigned_info": { "reason": "NODE_LEFT", "at": "2025-11-27T08:40:43.653Z", "details": "node_left [NKDiDmZ7TOShHAW32rcleg]", "last_allocation_status": "no_valid_shard_copy" }, "can_allocate": "no_valid_shard_copy", "allocate_explanation": "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster", "node_allocation_decisions": [ { "node_id": "WxxsBtxITtab58q078TdEg", "node_name": "opensearch-1.4c1", "transport_address": "10.45.114.208:9300", "node_attributes": { "app_id": "39b6cdac-c195-466d-8537-e4a1f41fafd0/opensearch", "shard_indexing_pressure_enabled": "true" }, "node_decision": "no", "store": { "found": false } }, { "node_id": "XnZt4LqwSTu79M7neGxkoQ", "node_name": "opensearch-0.4c1", "transport_address": "10.45.114.156:9300", "node_attributes": { "shard_indexing_pressure_enabled": "true", "app_id": "39b6cdac-c195-466d-8537-e4a1f41fafd0/opensearch" }, "node_decision": "no", "store": { "found": false } } ] } ``` Delete the problematic index identified in the previous step: #### WARNING If you do not have a snapshot containing this index, the data will be lost! ```shell curl --cacert cert.pem -X DELETE "https://:9200/index1" -u admin: ``` After deleting any orphaned indices, verify that the cluster returns to green or yellow health: ```shell curl --cacert cert.pem -X GET "https://:9200/_cluster/health?pretty" -u admin: ``` ### Set allocation settings During the upgrade process, the routing allocation setting may be restricted to `primaries`. Restore normal allocation by enabling all routing: ```shell curl --cacert cert.pem -X PUT "https://:9200/_cluster/settings" -H 'Content-Type: application/json' -u admin: -d' { "persistent": { "cluster.routing.allocation.enable": "all" } } ' ``` ### Add a new unit While optional, it is highly advisable to add a replacement unit to restore the application to its original scale: ```shell juju add-unit opensearch -n 1 ``` ### Remove rolled back unit Remove the rolled back unit: ```shell juju remove-unit opensearch/2 ``` Where `opensearch/2` is the name of the unit that was rolled back and blocked earlier. ### Remove lock If the replacement unit appears stuck displaying the status message `Requesting lock on operation: start`, check if the departed unit still hold the lock: ```text GET /.charm_node_lock/_doc/0 ``` Example response: ```json { "_index": ".charm_node_lock", "_id": "0", "_version": 3, "_seq_no": 28, "_primary_term": 1, "found": true, "_source": { "unit-name": "opensearch-2.4c1" } } ``` If the departed unit holds the lock, delete the lock document: ```shell curl --cacert cert.pem -X DELETE "https://:9200/.charm_node_lock/_doc/0?refresh=true" -u admin: ``` Wait for the replacement unit to start and join the cluster. `juju status` should show all units `active`/`idle` with no messages, and the application `active` with the original scale restored. ### Verify new unit has joined the cluster List the nodes in the current cluster: ```shell curl --cacert cert.pem -X GET "https://:9200/_cat/nodes" -u admin: ``` Confirm that the new node is present in the output, which will look similar to the following: ```text 10.45.114.228 35 86 6 0.43 0.69 0.95 dim cluster_manager,data,ingest,ml - opensearch-3.4c1 10.45.114.156 32 86 11 0.43 0.69 0.95 dim cluster_manager,data,ingest,ml * opensearch-0.4c1 10.45.114.208 45 86 11 0.43 0.69 0.95 dim cluster_manager,data,ingest,ml - opensearch-1.4c1 ``` See [mapping Juju units to OpenSearch nodes](https://canonical.com/data/opensearch/docs/2//explanation/cluster-health.md#cluster-health-mapping-nodes) for how to read this output. Finally, confirm the cluster is healthy again — the cluster health API should return `green`: ```shell curl --cacert cert.pem -XGET "https://:9200/_cluster/health?pretty" -u admin: ``` ## Next steps - [Back up and restore](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-guides-back-up-and-restore-index) — create a backup after upgrading. - [Scale a cluster horizontally](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally) — adjust cluster size if needed. # index.html.md # How to perform load testing This guide shows how to run load tests against a Charmed OpenSearch deployment using [opensearch-benchmark](https://github.com/opensearch-project/opensearch-benchmark). The example uses AWS, but the approach applies to any [Juju-supported cloud](https://juju.is/docs/juju/cloud). ## Prerequisites - Juju 3.6 (latest LTS) - [`jq`](https://jqlang.github.io/jq/) - A VPC on AWS (or equivalent in your cloud) - AWS `ACCESS_KEY` and `SECRET_KEY` - Optional, for the COS monitoring sections: a Kubernetes cluster (e.g. MicroK8s) with `kubectl` configured ## Set up the environment ### Set up Juju Define environment variables: ```shell export JUJU_CONTROLLER_NAME= export VPC_ID= export VPC_PRIVATE_CIDR= export K8S_CLOUD_NAME= ``` Add AWS credentials and bootstrap: ```shell juju add-credentials aws juju bootstrap aws $JUJU_CONTROLLER_NAME \ --credential aws_creds \ --model-default container-networking-method=local \ --config vpc-id=$VPC_ID \ --config vpc-id-force=true \ --constraints 'instance-type=t2.medium root-disk=100G' \ --to subnet=$VPC_PRIVATE_CIDR ``` ### Set up COS (optional, for monitoring) Deploy a K8s cluster (e.g. MicroK8s) and add it to Juju. See [COS installation guide](https://documentation.ubuntu.com/observability/track-2/tutorial/installation/cos-lite-microk8s-sandbox/#configure-microk8s). ```shell juju add-k8s $K8S_CLOUD_NAME --client --controller $JUJU_CONTROLLER_NAME juju add-model cos $K8S_CLOUD_NAME wget https://raw.githubusercontent.com/canonical/cos-lite-bundle/main/overlays/offers-overlay.yaml juju deploy cos-lite --trust --overlay ./offers-overlay.yaml ``` ### Access Grafana (optional) Retrieve the Traefik load balancer IP and the Grafana admin password: ```shell kubectl get svc -A | grep traefik | grep LoadBalancer | awk '{print $5}' juju run -m cos grafana/leader get-admin-password ``` Access Grafana at `http:///cos-grafana` (username: `admin`). ## Deploy OpenSearch Create a model with the required network and kernel configuration: ```shell juju add-model opensearch aws \ --config container-networking-method=local \ --config vpc-id=$VPC_ID \ --config vpc-id-force=true juju model-config cloudinit-userdata="postruncmd: - [ 'sysctl', '-w', 'vm.max_map_count=262144' ] - [ 'sysctl', '-w', 'fs.file-max=1048576' ] - [ 'sysctl', '-w', 'vm.swappiness=0' ]" juju add-space -m opensearch internal-space $VPC_PRIVATE_CIDR ``` Deploy OpenSearch with dedicated storage: ```shell juju deploy self-signed-certificates \ --constraints="arch=amd64 instance-type=t2.medium root-disk=100G spaces=internal-space" \ --bind "internal-space" juju create-storage-pool opensearch ebs volume-type=gp3 juju deploy opensearch \ --channel=2/stable --config profile=production -n3 \ --constraints="arch=amd64 instance-type=r5.xlarge root-disk=200G spaces=internal-space" \ --bind "internal-space" \ --storage opensearch-data=opensearch,512G juju integrate opensearch:certificates self-signed-certificates ``` ## Integrate with COS (optional) Consume the COS offers and integrate Grafana Agent with OpenSearch: ```shell juju consume admin/cos.grafana-dashboards juju consume admin/cos.loki-logging juju consume admin/cos.prometheus-receive-remote-write juju deploy grafana-agent juju integrate grafana-agent opensearch:cos-agent juju integrate grafana-agent grafana-dashboards juju integrate grafana-agent loki-logging juju integrate grafana-agent prometheus-receive-remote-write ``` ## Run the benchmark Retrieve the admin password and save the CA certificate chain so that the benchmark client can verify the TLS certificates OpenSearch serves: ```shell juju run opensearch/leader get-password export OPENSEARCH_PWD= juju run opensearch/leader get-password --format=json \ | jq -r '.[].results."ca-chain"' > cert.pem ``` Install opensearch-benchmark: ```shell pip install opensearch-benchmark ``` Build the host list and execute: ```shell export OPENSEARCH_HOSTS="$(juju status --format=json | jq -r '.applications.opensearch.units[] | ."public-address"' | sed 's/.*/https:\/\/&:9200/' | paste -s -d, -)" opensearch-benchmark run \ --target-hosts $OPENSEARCH_HOSTS \ --pipeline benchmark-only \ --workload nyc_taxis \ --client-options basic_auth_user:admin,basic_auth_password:$OPENSEARCH_PWD,verify_certs:false ``` See the [opensearch-benchmark documentation](https://opensearch.org/docs/2.19/benchmark/) for additional workloads and options. When the benchmark completes, it prints a summary report with throughput, latency, and error metrics. If COS is integrated, the Grafana **Charmed OpenSearch** dashboard shows the load spike during the test. ## Next steps - [Optimize cluster performance with profiles](https://canonical.com/data/opensearch/docs/2//how-to/optimize-cluster-performance.md#how-to-optimize-cluster-performance) — tune resource allocation based on benchmark results. - [Enable monitoring (COS)](https://canonical.com/data/opensearch/docs/2//how-to/monitoring.md#how-to-monitoring) — set up ongoing observability. # index.html.md # How to optimize cluster performance with profiles This guide shows how to configure performance profiles for Charmed OpenSearch to match your workload requirements. Charmed OpenSearch supports two profiles: - **`testing`** — lightweight workloads (1 GB heap, minimum 1 node) - **`production`** — production workloads (50% RAM heap, minimum 3+3 nodes) ## Deploy with the `testing` profile `testing` is the default profile, so it applies when `profile` is not set. To set it explicitly: ```shell juju deploy opensearch --channel=2/stable --config profile=testing ``` The `testing` profile allows a single node and sets JVM heap to 1 GB. TLS is still required. #### NOTE A warning will appear in logs indicating the testing profile is active. This profile is not suitable for production. ## Deploy with the `production` profile Deploy a multi-unit cluster with the production profile: ```shell juju deploy opensearch --channel=2/stable --config profile=production -n 3 ``` The `production` profile enforces: - Minimum 3 cluster manager nodes - Minimum 3 data nodes - Minimum memory: 8 GB per node - JVM heap: 50% of available RAM (minimum 4 GB, maximum 31 GB) If requirements are not met, the charm remains `blocked` until corrected. ## Change the profile at runtime The profile can be changed after deployment: ```shell juju config opensearch profile= ``` Where `` is `testing` or `production`. The charm automatically reconfigures the cluster to match the new profile requirements. Once complete, `juju status` shows the OpenSearch application `active`. ## Profile comparison For a detailed explanation of what profiles are and why they exist, see [Performance profiles](https://canonical.com/data/opensearch/docs/2//explanation/performance-profiles.md#explanation-performance-profiles). | Setting | `testing` | `production` | |--------------------|-----------------------|------------------------------------------| | Cluster size | Minimum 1 node | Minimum 3 cluster manager + 3 data nodes | | Memory requirement | None enforced | Minimum 8 GB per node | | JVM heap | Fixed 1 GB | 50% of RAM (min 4 GB, max 31 GB) | | Use case | Development / testing | Production workloads | ## Next steps - [Standard deployment](https://canonical.com/data/opensearch/docs/2//how-to/deploy/standard.md#how-to-deploy-standard) — deploy a cluster with a specific profile. - [Scale a cluster horizontally](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally) — adjust cluster size after changing the profile. # index.html.md # How to enable email notifications This guide shows how to configure SMTP credentials so that OpenSearch can send email notifications via the [SMTP integrator charm](https://charmhub.io/smtp-integrator). #### NOTE SMTP integration requires Charmed OpenSearch revision 344 or higher. #### NOTE SMTP servers using self-signed certificates are not currently supported. ## Deploy and configure the SMTP integrator Deploy the charm: ```shell juju deploy smtp-integrator --channel latest/edge ``` The charm will be `blocked` until configured. Provide SMTP credentials: ```shell juju config smtp-integrator \ host= \ port=587 \ user= \ password= \ smtp_sender= \ recipients=, ``` ## Integrate with OpenSearch Connect the SMTP integrator to OpenSearch: ```shell juju integrate smtp-integrator:smtp opensearch:smtp ``` Once the integration is established, `juju status --relations` shows both applications `active` with an `smtp` relation between them, and OpenSearch can send email notifications via the configured SMTP server. #### NOTE OpenSearch requires a stable identifier for each email sender, notification channel, and notification group. The charm derives this identifier from the Juju relation ID of the SMTP integration, so the relation ID must remain stable. If you remove and re-add the integration, OpenSearch treats each reconnection as a new sender and channel, and the previous entries become stale. See the [OpenSearch Notifications documentation](https://docs.opensearch.org/2.19/observing-your-data/notifications/index/#create-email-sender) for details. ### Large deployments In large deployments, the SMTP integrator must be integrated with the **main orchestrator** application. Identify the main orchestrator by inspecting `juju status` integrations: ```text Integration provider Requirer Interface Type Message opensearch-main:peer-cluster-orchestrator opensearch-data:peer-cluster peer_cluster regular ``` Integrate with the main orchestrator: ```shell juju integrate smtp-integrator:smtp opensearch-main ``` If integrated with the wrong application, the charm shows a `blocked` status. Remove the invalid relation and integrate with the correct application. ## Next steps - [Enable monitoring (COS)](https://canonical.com/data/opensearch/docs/2//how-to/monitoring.md#how-to-monitoring) — set up alerting and observability for the cluster. - [Access OpenSearch using OAuth](https://canonical.com/data/opensearch/docs/2//how-to/access-using-oauth.md#how-to-access-using-oauth) — configure authentication for client access. # index.html.md # How to manage passwords This guide shows how to retrieve and rotate the admin password and how to rotate the credentials Charmed OpenSearch issues to integrated client applications. ## Manage the admin password The `admin` user is an internal charm user. Use its credentials for cluster administration only — client applications must obtain their own credentials through an integration. #### NOTE The `get-password` and `set-password` actions take an optional `username` parameter. Besides `admin` (the default), they accept other internal charm users: `kibanaserver` (used by OpenSearch Dashboards) and `monitor` (used by COS monitoring). `set-password` must be run on the leader unit. ### Retrieve the admin password To get admin password: ```shell juju run opensearch/leader get-password ``` The action returns the `admin` password and the CA certificate chain used to generate the admin client certificate. To retrieve the password of another internal user, pass its username: `juju run opensearch/leader get-password username=`. ### Rotate the admin password To generate a random password: ```shell juju run opensearch/leader set-password ``` To set a specific password: ```shell juju run opensearch/leader set-password password= ``` Both commands return the new password as `admin-password`. The previous password stops working immediately. To rotate the password of another internal user, pass its username: ```shell juju run opensearch/leader set-password username= password= ``` ## Manage client credentials Charmed OpenSearch generates a dedicated user and password for each client integration. These credentials are passed to the client application over the relation, so you do not retrieve or set them directly. ### Rotate client credentials Remove and re-add the integration to generate a new user with a new password: ```shell juju remove-relation opensearch juju integrate opensearch ``` The client application receives the new credentials over the relation. ## Next steps - [Manage TLS encryption](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-enable-tls-encryption) to rotate certificates and private keys. - [Integrate with an application](https://canonical.com/data/opensearch/docs/2//how-to/integrate-with-an-application.md#how-to-integrate-with-an-application) to connect a client. # index.html.md # How to manage TLS encryption This guide shows how to enable TLS encryption, update private keys, and rotate TLS/CA certificates for a Charmed OpenSearch deployment. For a step-by-step introduction, see the [Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/3-enable-encryption.md#tutorial-3-enable-encryption). #### NOTE TLS is mandatory for Charmed OpenSearch and cannot be disabled. ## Enable TLS encryption The example below uses the [`self-signed-certificates` operator](https://github.com/canonical/self-signed-certificates-operator). #### CAUTION **[Self-signed certificates](https://en.wikipedia.org/wiki/Self-signed_certificate) are not recommended for production.** See the [X.509 certificates topic](https://charmhub.io/topics/security-with-x-509-certificates) for an overview of available certificate charms. Deploy the TLS charm: ```shell juju deploy self-signed-certificates --config ca-common-name="My CA" ``` Integrate it with OpenSearch: ```shell juju integrate opensearch:certificates self-signed-certificates ``` Verify the relation with `juju status --relations`. The OpenSearch application turns `active` — it is no longer `blocked` with a “Missing TLS relation” message. ## Check certificates in use To inspect the issuer of the certificate currently served by OpenSearch: ```shell openssl s_client -showcerts -connect : < /dev/null | grep issuer ``` ## Update private keys Private keys can be updated via the `set-tls-private-key` action. Charmed OpenSearch uses three certificate categories: - `app-admin` — administrative actions (leader unit only) - `unit-transport` — internal node-to-node communication - `unit-http` — external client-to-node communication #### NOTE Each unit has its own node certificate, so `unit-transport` and `unit-http` keys must be set on every target unit individually (e.g. `juju run opensearch/ ...`). For an explanation of why these three categories exist and how rotation works, see [TLS certificates](https://canonical.com/data/opensearch/docs/2//explanation/security/tls-certificates.md#explanation-tls-certificates). ### Auto-generate new keys Generate the `app-admin` key on the leader: ```shell juju run opensearch/leader set-tls-private-key category=app-admin ``` Then generate the `unit-transport` and `unit-http` keys on **each unit**, replacing `` (for example, `opensearch/0`, `opensearch/1`): ```shell juju run opensearch/ set-tls-private-key category=unit-transport juju run opensearch/ set-tls-private-key category=unit-http ``` ### Use custom keys Generate keys with OpenSSL: ```shell openssl genrsa -out unit-http.pem 3072 openssl genrsa -out unit-transport.pem 3072 openssl genrsa -out app-admin.pem 3072 ``` Apply the `app-admin` key on the leader: ```shell juju run opensearch/leader set-tls-private-key category=app-admin key="$(base64 -w0 app-admin.pem)" ``` Then apply the `unit-transport` and `unit-http` keys on **each unit**, replacing `` (for example, `opensearch/0`, `opensearch/1`): ```shell juju run opensearch/ set-tls-private-key category=unit-transport key="$(base64 -w0 unit-transport.pem)" juju run opensearch/ set-tls-private-key category=unit-http key="$(base64 -w0 unit-http.pem)" ``` ## Rotate TLS certificates Certificate rotation is triggered automatically when a certificate expires. To rotate manually, regenerate the private key for the desired category: ```shell juju run opensearch/ set-tls-private-key category= ``` Where `` is `app-admin`, `unit-transport`, or `unit-http`. Use `opensearch/leader` for `app-admin`; use `opensearch/` for `unit-transport` and `unit-http`, repeating for each unit. This generates a new private key and CSR, which is sent to the certificate provider for signing. Once signed, the new certificate is automatically applied to the cluster. ## Rotate CA certificates The CA certificate is provided by the TLS operator you are using. The rotation process differs depending on the operator. ### With `self-signed-certificates` Trigger CA rotation by changing the common name: ```shell juju config self-signed-certificates ca-common-name= ``` This will: 1. Generate a new CA certificate with the new common name. 2. Revoke all previously issued TLS certificates. 3. Cause OpenSearch to automatically request new certificates. 4. Trigger a rolling restart to apply the new CA across all nodes. Until the rolling restart completes, nodes continue using the old certificates. Verify the new CA is in use: ```shell openssl s_client -showcerts -connect : < /dev/null | grep issuer ``` ### With `manual-tls-certificates` To rotate the CA with the `manual-tls` operator, sign CSRs with the new CA certificate and provide them to the cluster. If you no longer have the original CSR files, [regenerate them]() first. Provide the new certificate to each unit, **starting with the leader**: ```shell juju run manual-tls-certificates/leader provide-certificate \ relation-id= \ certificate="$(base64 -w0 certificate.pem)" \ ca-chain="$(base64 -w0 ca_chain.pem)" \ ca-certificate="$(base64 -w0 ca_certificate.pem)" \ certificate-signing-request="$(base64 -w0 csr.pem)" \ unit-name="" ``` #### CAUTION Always distribute certificates to the leader unit first, then to the remaining nodes. After receiving the new CA, each node generates new CSRs that must be signed with the new CA and provided back. Repeat for every unit in the cluster. Once all units have the new CA, OpenSearch reloads certificates (or triggers a rolling restart if the issuer, subject, or subject alternative names (SANs) have changed). Verify the rotation: ```shell openssl s_client -showcerts -connect : < /dev/null | grep issuer ``` ## Next steps - [Enable monitoring (COS)](https://canonical.com/data/opensearch/docs/2//how-to/monitoring.md#how-to-monitoring) — observe the cluster with TLS in place. - [Security explanation](https://canonical.com/data/opensearch/docs/2//explanation/security/index.md#explanation-security-index) — background on TLS and authentication. # index.html.md # How to enable LDAP authentication LDAP (Lightweight Directory Access Protocol) enables centralized authentication for OpenSearch clusters, reducing the overhead of managing local credentials and access policies. This guide shows how to integrate LDAP as an authentication method with Charmed OpenSearch. ## Prerequisites ### VM You’ll need: - Charmed OpenSearch - A Kubernetes cloud registered with your Juju controller ### K8s You’ll need: - Charmed OpenSearch K8s ## Deploy an LDAP server on Kubernetes Charmed OpenSearch works with any provider of the `ldap` interface. This guide covers the two Canonical-maintained ones: the [Authentik LDAP outpost](https://charmhub.io/authentik-ldap-outpost), which exposes an Authentik directory over LDAP, and [GLAuth](https://charmhub.io/glauth-k8s), a lightweight LDAP server. Both are Kubernetes charms. ### VM For machine deployments, create a Kubernetes model for the LDAP provider and its supporting applications. We’ll then create a cross-model relation to the OpenSearch VM model. ```shell juju add-model ``` ### K8s For Kubernetes deployments, you can simply deploy the LDAP provider and its supporting applications alongside OpenSearch without a separate Juju model. OpenSearch only connects to LDAP over LDAPS. This example uses `self-signed-certificates` as the certificate provider. If it is already deployed in the model containing the LDAP provider, reuse it. Otherwise, deploy it: ```shell juju deploy self-signed-certificates --channel 1/stable ``` #### NOTE This guide uses `self-signed-certificates` for demonstration. For production deployments, use your organisation’s preferred certificate authority. ### Authentik Deploy the Authentik server, its worker, the LDAP outpost, and the supporting database and ingress charms: ```shell juju deploy postgresql-k8s --channel 14/stable --trust juju deploy authentik-server --channel latest/stable --trust juju deploy authentik-worker --channel latest/stable --trust juju deploy authentik-ldap-outpost --channel latest/stable --trust juju deploy traefik-k8s --trust ``` Integrate them: ```shell juju integrate authentik-server:pg-database postgresql-k8s:database juju integrate authentik-server:authentik-cluster authentik-worker juju integrate authentik-server:authentik-server-info authentik-ldap-outpost juju integrate authentik-server:traefik-route traefik-k8s juju integrate authentik-ldap-outpost:traefik-route traefik-k8s ``` Traefik is required: the outpost does not provision a certificate of its own, so Traefik terminates LDAPS for it on port 636. Wait until `traefik-k8s` is `active`, then provide it with a certificate: ```shell juju integrate traefik-k8s:certificates self-signed-certificates ``` #### NOTE Traefik requests its certificate for its load balancer address. If the integration is added before that address is assigned, Traefik keeps serving its default certificate, which OpenSearch does not trust. The outpost serves the directory under the base DN `dc=ldap,dc=goauthentik,dc=io`. To use a different one, set the `base_dn` config option of `authentik-ldap-outpost`. Users and groups are managed in Authentik itself. To reach its web interface, generate a recovery link for the bootstrap administrator: ```shell juju run authentik-server/leader create-recovery-link ``` #### NOTE The outpost defaults to `search_mode=cached` and `bind_mode=cached`, which serve a periodically refreshed snapshot of the directory. Directory changes – new users, changed group membership, changed passwords – can take until the next synchronisation to reach OpenSearch. Set both to `direct` if every request must reflect the live Authentik state. ### GLAuth Deploy `glauth-k8s` and `postgresql-k8s`: ```shell juju deploy glauth-k8s --channel latest/stable --trust --config ldaps_enabled=true juju deploy postgresql-k8s --channel 14/stable --trust ``` `ldaps_enabled=true` is **required**: GLAuth starts with LDAPS disabled. Integrate `glauth-k8s` with `self-signed-certificates` and `postgresql-k8s`: ```shell juju integrate glauth-k8s self-signed-certificates juju integrate glauth-k8s:pg-database postgresql-k8s:database ``` Deploy the [Traefik charm](https://charmhub.io/traefik-k8s) to expose the LDAPS endpoint of GLAuth, and integrate the two applications: ```shell juju deploy traefik-k8s --trust juju integrate traefik-k8s glauth-k8s:ldaps-ingress ``` Deploy the [`glauth-utils` charm](https://charmhub.io/glauth-utils) to manage LDAP users, and integrate it with the GLAuth application: ```shell juju deploy glauth-utils --channel latest/stable --trust juju integrate glauth-k8s glauth-utils ``` Users and groups can now be created using `glauth-utils`. ## Create a cross-model relation (VM only) ### VM ### Expose cross-model relations Offer the `ldap` endpoint of the provider you deployed – `authentik-ldap-outpost` or `glauth-k8s` – together with the CA certificate that signed its LDAPS certificate: ```shell juju offer :ldap ldap juju offer self-signed-certificates:send-ca-cert ldap-cert ``` ### Consume offers Switch to the OpenSearch model: ```shell juju switch ``` Consume the LDAP offers: ```shell juju consume admin/.ldap juju consume admin/.ldap-cert ``` The OpenSearch machines must be able to reach the address that Traefik exposes. To see it, run `juju run traefik-k8s/0 show-proxied-endpoints` in the Kubernetes model. ### K8s This procedure applies to machine deployments. On Kubernetes, proceed to the next section: [Configure roles](#ldap-configure-roles). ## Configure roles Permissions are granted to an LDAP group through the [`data-integrator` charm](https://charmhub.io/data-integrator). The LDAP group name **must be identical** to the key of the Juju secret created below. Juju secret keys must: - start with a lowercase letter - be at least three characters long - contain only lowercase letters, digits and single hyphens ### Grant permissions to a group Make sure the model containing OpenSearch is selected: ```shell juju switch ``` Deploy one `data-integrator` for each LDAP group you want to grant permissions to: ```shell juju deploy data-integrator search-admin-di --channel latest/edge ``` Create a Juju secret whose key is the LDAP group name, and grant it to the integrator: ```shell juju add-secret search-admin-secret search-admin=password juju grant-secret search-admin-secret search-admin-di ``` Configure the `data-integrator`: ```shell juju config search-admin-di \ index-name=search-index \ entity-type=GROUP \ entity-permissions='[{"resource_name":"search-index","resource_type":"index_permissions","privileges":["read","search","get","write"]}]' \ requested-entities-secret= ``` - `entity-type`: must be `GROUP` to grant permissions to an LDAP group. - `entity-permissions`: a list with one entry containing: - `resource_name`: the index or index pattern the group may access. - `resource_type`: must be `index_permissions` - `privileges`: the OpenSearch action groups to allow. - `requested-entities-secret`: the URI of the secret whose key is the group name. Integrate it with OpenSearch: ```shell juju integrate search-admin-di opensearch ``` ### Create the group and its users in LDAP Make sure the LDAP group exists and contains its users. Its name must be identical to the key of the Juju secret above. ### Authentik Create the group and its users in the Authentik web interface, and add the users to the group. ### GLAuth Create the group and its users with `glauth-utils`. ### Configure the directory layout OpenSearch uses four config options to search the directory: - `ldap_user_base`: The base DN in the directory tree from which user searches are performed. Default `ou=users,dc=glauth,dc=com` - `ldap_user_search`: The search filter used to locate a user entry, where `{0}` is replaced with the login username. Default `(cn={0})` - `ldap_user_rolename`: The attribute on a user entry that specifies which groups the user belongs to. Default `memberOf` - `ldap_role_name_attr`: The attribute on a group entry that OpenSearch will read as the group name. Default `ou` ### Authentik Authentik stores users under `ou=users` of its base DN and names groups with a `cn` RDN. Point OpenSearch at them, substituting the base DN if you changed it: ```shell juju config opensearch \ ldap_user_base="ou=users,dc=ldap,dc=goauthentik,dc=io" \ ldap_role_name_attr=cn ``` ### GLAuth The defaults match the layout of GLAuth. If your directory uses a different schema, adjust the options above. ## Enable LDAP To enable LDAP authentication, integrate the OpenSearch charm with the LDAP provider and with the CA certificate of its LDAPS endpoint: ### VM ```shell juju integrate opensearch:ldap ldap juju integrate opensearch:ldap-certificate-transfer ldap-cert ``` ### K8s Substitute `` with `authentik-ldap-outpost` or `glauth-k8s`: ```shell juju integrate opensearch:ldap :ldap juju integrate opensearch:ldap-certificate-transfer self-signed-certificates:send-ca-cert ``` Check the status of the OpenSearch model: ```shell juju status -m ``` Wait until all applications report `active` and all unit agents report `idle`. When everything has stabilised, LDAP users can log in with their directory credentials and inherit the permissions granted by the roles corresponding to their LDAP groups. ## Disable LDAP You can disable LDAP by removing the relations with the LDAP provider: ### VM ```shell juju remove-relation opensearch:ldap-certificate-transfer ldap-cert juju remove-relation opensearch:ldap ldap ``` ### K8s Substitute `` with `authentik-ldap-outpost` or `glauth-k8s`: ```shell juju remove-relation opensearch:ldap-certificate-transfer self-signed-certificates:send-ca-cert juju remove-relation opensearch:ldap :ldap ``` The directory layout options keep their values after the relations are removed. Reset them before integrating a different LDAP provider: ```shell juju config opensearch --reset ldap_user_base,ldap_user_search,ldap_user_rolename,ldap_role_name_attr ``` # index.html.md # How to back up and restore This guide shows how to create backups (snapshots) of a Charmed OpenSearch cluster, restore from a backup, and migrate data to a new cluster. ## Prerequisites - A cluster with at least three nodes deployed and `active` - Access to S3-compatible, Azure, or Google Cloud Storage (see [Configure S3](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-s3.md#how-to-back-up-configure-s3), [Configure Azure storage](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-azure-storage.md#how-to-back-up-configure-azure-storage), or [Configure Google Cloud Storage](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-gcs-storage.md#how-to-back-up-configure-gcs-storage)) - Storage integration already established with OpenSearch #### CAUTION Only one object storage integrator can be related at a time. Relating more than one places OpenSearch in a `blocked` state until you remove the extra relations. ## Save cluster credentials Backups exclude the security configuration, so the source cluster’s users, passwords, and role mappings cannot be restored. Before you run a restore or migrate to a new cluster, save the admin password and the CA certificates of the target cluster: ```shell juju run opensearch/leader get-password ``` #### CAUTION Excluding the security configuration does not protect the data itself. The snapshot repository still contains all indexed application data in an unencrypted form. Anyone with access to the repository can register it in another compatible OpenSearch cluster and restore that data using their own cluster’s admin credentials. Protect the repository with strict access controls and encryption at rest. ## Create a backup Confirm the cluster is `active` and `idle` with `juju status`, then run: ```shell juju run opensearch/leader create-backup ```
Output example ```text backup-id: "2026-01-01t10:30:00z" status: in_progress ```
The action only *initiates* the snapshot; it does not wait for it to finish. To confirm completion: 1. Record the `backup-id` returned by the action. 2. Run [`list-backups`](). 3. Repeat until that `backup-id` shows a `success` status. Track the backup by its `backup-id`, not by position in the list: other snapshots can be created directly through the OpenSearch API, so the newest entry is not necessarily yours. #### CAUTION Never restore or migrate a backup that is `in_progress` or `failed`. Only a `success` backup is complete and safe to use. ## List backups To list available, failed, and in-progress backups: ```shell juju run opensearch/leader list-backups ```
Output example ```text backup-id | backup-status ------------------------------------- 2026-01-01t10:30:00z | in_progress 2026-01-01t10:00:00z | success 2026-01-01t09:00:00z | failed ```
## Restore a backup To restore a backup that was made from a different cluster (cluster migration), see [Migrate to a new cluster]() below. To restore from the same cluster, pass the `backup-id` from `list-backups`: ```shell juju run opensearch/leader restore backup-id= ``` After the restore completes, `juju status` shows the OpenSearch application `active` and the cluster health API returns `green`. #### NOTE If the restore takes longer than the Juju CLI timeout, it continues in the background. Monitor progress with `juju status`. ## Migrate to a new cluster To migrate data from one cluster to another, configure the new cluster to use the same storage backend where the old cluster’s backups reside, then restore: ```shell juju run opensearch/leader restore backup-id= ``` The `` must reference a backup created by the previous cluster. ## Next steps - [Upgrade, rollback, and recover](https://canonical.com/data/opensearch/docs/2//how-to/upgrade.md#how-to-minor-upgrade) — upgrade the cluster after restoring. - [Manage persistent storage](https://canonical.com/data/opensearch/docs/2//how-to/persistent-storage.md#how-to-persistent-storage) — reuse disks when no viable snapshot exists. # index.html.md # How to configure Azure storage This guide shows how to configure the [Azure Storage Integrator charm](https://charmhub.io/azure-storage-integrator) for OpenSearch backups. ## Deploy and configure the integrator Deploy the charm and set storage account details: ```shell juju deploy azure-storage-integrator --channel latest/edge juju config azure-storage-integrator \ storage-account= \ container= ``` Provide the storage key via a Juju secret: ```shell juju add-secret azure-secret secret-key= juju grant-secret azure-secret azure-storage-integrator juju config azure-storage-integrator credentials= ``` See the [azure-storage-integrator configuration reference](https://charmhub.io/azure-storage-integrator/configuration) for all available options. ## Integrate with Charmed OpenSearch Connect the integrator to OpenSearch: ```shell juju integrate azure-storage-integrator opensearch ``` Once the integration is established, `juju status --relations` shows the `azure-storage-integrator` application `active` with a relation to `opensearch`. The OpenSearch application remains `active`. ## Next steps - [Create and restore backups](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-create-a-backup) — create a backup using the configured Azure storage. - [Configure S3 storage](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-s3.md#how-to-back-up-configure-s3) — alternative storage backend. # index.html.md # How to configure Google Cloud Storage This guide shows how to configure the [GCS Integrator charm](https://charmhub.io/gcs-integrator) for OpenSearch backups. ## Prepare the Google Cloud service account In your Google Cloud project, create a service account and download its JSON key: ```shell gcloud iam service-accounts create opensearch-backup gcloud iam service-accounts keys create service-account.json \ iam.gserviceaccount.com/opensearch-backup@.iam.gserviceaccount.com ``` Grant the service account permission to list, read, and write objects in the bucket: ```shell gcloud projects add-iam-policy-binding \ --member=serviceAccount:opensearch-backup@.iam.gserviceaccount.com \ --role=roles/storage.objectAdmin ``` #### NOTE If the bucket does not exist yet, Charmed OpenSearch creates it during credential verification, which also requires permission to create buckets in the project (`roles/storage.admin`). Bucket names in Google Cloud Storage are globally unique. ## Deploy and configure the integrator Deploy the charm: ```shell juju deploy gcs-integrator --channel 1/stable ``` Store the service-account key in a [Juju secret](https://canonical-juju.readthedocs-hosted.com/en/latest/user/reference/secret/) and grant access to the integrator: ```shell juju add-secret gcs-secret secret-key#file=service-account.json juju grant-secret gcs-secret gcs-integrator ``` The `#file=` suffix tells Juju to read the value from the local file rather than from the command-line argument, keeping the private key out of your shell history. #### NOTE Charmed OpenSearch accepts service-account keys as plain JSON or base64-encoded JSON, so no decoding is required. Point the integrator at the secret and configure the bucket: ```shell juju config gcs-integrator \ credentials=secret: \ bucket= \ path= ``` See the [gcs-integrator configuration reference](https://charmhub.io/gcs-integrator/configuration) for all available options. ## Integrate with Charmed OpenSearch Connect the integrator to OpenSearch: ```shell juju integrate gcs-integrator opensearch ``` Once the integration is established, `juju status --relations` shows the `gcs-integrator` application `active` with a `gcs-credentials` relation to `opensearch`. The OpenSearch application remains `active`. #### CAUTION Only one object storage integrator can be related at a time. Relating both the GCS integrator and the S3 or Azure Storage integrator places OpenSearch in a `blocked` state with the message `Too many object storage relations. Only one is supported.` until you remove the extra relations. ## Next steps - [Create and restore backups](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-create-a-backup) — create a backup using the configured Google Cloud Storage. - [Configure S3 storage](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-s3.md#how-to-back-up-configure-s3) — alternative storage backend. - [Configure Azure storage](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-azure-storage.md#how-to-back-up-configure-azure-storage) — alternative storage backend. # index.html.md # How to configure S3 storage This guide shows how to configure the [S3 Integrator charm](https://charmhub.io/s3-integrator) for OpenSearch backups, using either AWS S3 or Ceph RadosGW. ## Configure S3 for AWS Deploy the `s3-integrator` charm and provide credentials: ```shell juju deploy s3-integrator juju run s3-integrator/leader sync-s3-credentials \ access-key= secret-key= ``` Configure the bucket and endpoint: ```shell juju config s3-integrator \ bucket= \ region= \ endpoint=s3..amazonaws.com ``` #### NOTE The endpoint must be specified as `s3..amazonaws.com` within the first 24 hours of creating the bucket. For older buckets, `s3.amazonaws.com` can be used. See [this AWS knowledge centre article](https://repost.aws/knowledge-center/s3-http-307-response). See the [s3-integrator configuration reference](https://charmhub.io/s3-integrator/configuration) for all available options. ## Configure S3 for Ceph RadosGW Deploy the `s3-integrator` charm and provide credentials: ```shell juju deploy s3-integrator juju run s3-integrator/leader sync-s3-credentials \ access-key= secret-key= ``` Configure the endpoint and bucket: ```shell juju config s3-integrator \ endpoint="https://" \ bucket= \ path= \ region= \ tls-ca-chain="$(base64 -w0 )" ``` `tls-ca-chain` is the base64-encoded CA certificate that signed the RadosGW TLS certificate. #### NOTE Set `region` to `default` if no custom region was configured in Ceph. See the [RadosGW documentation](https://docs.ceph.com/en/latest/man/8/radosgw-admin/) for details. ## Integrate with Charmed OpenSearch Connect the s3-integrator to OpenSearch: ```shell juju integrate s3-integrator opensearch ``` Once the integration is established, `juju status --relations` shows the `s3-integrator` application `active` with an `s3-credentials` relation to `opensearch`. The OpenSearch application remains `active`. ## Next steps - [Create and restore backups](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-create-a-backup) — create a backup using the configured S3 storage. - [Configure Azure storage](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-azure-storage.md#how-to-back-up-configure-azure-storage) — alternative storage backend. # index.html.md # How to deploy These guides cover deploying Charmed OpenSearch in different environments. For a step-by-step introduction, see the [Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/index.md#tutorial-index). * [Standard deployment](https://canonical.com/data/opensearch/docs/2//how-to/deploy/standard.md) * [Large deployment](https://canonical.com/data/opensearch/docs/2//how-to/deploy/large.md) # index.html.md # How to deploy Charmed OpenSearch This guide walks you through deploying Charmed OpenSearch, covering both the **IAAS/VM** charm (`opensearch`) and the **Kubernetes** charm (`opensearch-k8s`). If you are new to OpenSearch or Juju and are looking for a more comprehensive walkthrough of these steps, see the [Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/index.md#tutorial-index). For large, multi-application deployments, see the [Launch a large deployment](https://canonical.com/data/opensearch/docs/2//how-to/deploy/large.md#how-to-deploy-large) guide instead. ## Prerequisites Check that you fulfill the hardware requirements in the [system requirements page](https://canonical.com/data/opensearch/docs/2//reference/system-requirements.md#reference-system-requirements). Before continuing, decide whether you are going to use a machine (VM)-based or a Kubernetes environment for this deployment. Use the tabs below to switch between the two substrates. The instructions will update accordingly. ### VM To deploy Charmed OpenSearch using Juju in machine/VM environment, you need: * **Juju `3.6+` (latest LTS)** – Canonical’s orchestration engine (see [How to install Juju](https://canonical.com/juju/docs/juju-cli/3.6/howto/manage-juju/#install-juju)) * **LXD `v6.1+`** – Canonical’s lightweight container hypervisor (see [LXD tutorial](https://canonical.com/lxd/docs/latest/tutorial/first_steps/#install-lxd-using-snap)). ### K8s To deploy Charmed OpenSearch using Juju in K8s environment, you need: * **Juju `3.6+` (latest LTS)** – Canonical’s orchestration engine (see [How to install Juju](https://canonical.com/juju/docs/juju-cli/3.6/howto/manage-juju/#install-juju)) * **Kubernetes `v1.29+` cluster**, for example: * [Canonical Kubernetes](https://documentation.ubuntu.com/canonical-kubernetes/latest/) with the following features enabled: * `local-storage` * `load-balancer` * [MicroK8s](https://canonical.com/microk8s/docs/getting-started) with the following add-ons: * `hostpath-storage` * `dns` * `metallb` ## Prepare the substrate Prepare the environment for Charmed OpenSearch deployment: ### VM **Disable IPv6 on LXD** Juju does not support IPv6 addresses with LXD. To set the network bridge to have no IPv6 addresses, run the following command after initializing LXD: ```shell lxc network set lxdbr0 ipv6.address none ``` See [The LXD cloud and Juju](https://canonical.com/juju/docs/juju-cli/3.6/reference/cloud/list-of-supported-clouds/lxd/#constraints) for more information. ### K8s **Check the storage class** Charmed OpenSearch K8s requests two persistent volumes per unit. Confirm that your cluster has a storage class that can satisfy them: ```shell kubectl get storageclass ``` The controller bootstrap and model-creation commands below assume a suitable storage class is marked `(default)`. If your cluster has no default, see [Juju’s Kubernetes storage configuration](https://canonical.com/juju/docs/juju-cli/3.6/reference/cloud/list-of-supported-clouds/microk8s/#models) before continuing. ## Bootstrap a Juju controller Make sure your cloud is registered with Juju: ```shell juju list-clouds ``` #### NOTE See also: [How to manage clouds](https://canonical.com/juju/docs/juju-cli/latest/howto/manage-clouds/) in the Juju documentation. Bootstrap a new controller: ```shell juju bootstrap ``` Or switch to an existing one: ```shell juju switch ``` ### VM Make sure that the controller’s back-end cloud is **not** Kubernetes-based. ### K8s Make sure that the controller’s back-end cloud **is** Kubernetes-based. ## Create a model Create a model if you haven’t already: ```shell juju add-model ``` If you are reusing an existing model, select it before running the remaining commands: ```shell juju switch : ``` Check that the model is of the expected type: ```shell juju show-model ``` The output includes a `model-type` field (distinct from the cloud’s `type` field). ### VM The `model-type` must **not** be `caas`. ### K8s The `model-type` must be `caas`. ## Kernel parameter configuration OpenSearch relies on a number of kernel parameters that are not set to suitable values by default. Configure them before deploying OpenSearch. How and where you apply them depends on the substrate. #### NOTE To take note of the current values before changing them: ```shell sudo sysctl -a | grep -E 'swappiness|max_map_count|file-max' ``` The settings below are saved in `/etc/sysctl.d/opensearch.conf` and persist across reboots. To restore the previous values, remove or update that file and reset the parameters manually, or reboot after removing the file. ### VM Configure the required kernel settings on the host machine, then configure the workload model to apply them to new containers. You can do this after bootstrapping a controller, but before deploying OpenSearch. **Configure sysctl on the host machine** On the **host** machine, run the following command to add the settings to a config file: ```shell sudo tee /etc/sysctl.d/opensearch.conf < cloudinit-userdata.yaml cloudinit-userdata: | postruncmd: - echo 'vm.max_map_count=262144' >> /etc/sysctl.conf - echo 'vm.swappiness=0' >> /etc/sysctl.conf - echo 'fs.file-max=1048576' >> /etc/sysctl.conf - sysctl -p EOF ``` #### NOTE Keep each `postruncmd` entry as a **string**. Cloud-init runs string entries through a shell, so the `>>` redirection works. Entries written as a YAML list are passed straight to `execve(3)` with no shell, so `>>` would become a literal argument to `echo` instead of appending to the file. Apply it to the **existing model** before deploying OpenSearch, so the settings are included when Juju provisions its machines. Changing model configuration does not retroactively run `cloud-init` on existing machines: ```shell juju model-config --file=./cloudinit-userdata.yaml --model ``` For models you create **in the future**, you can instead set this as a default on the selected controller using [`juju model-defaults`](https://canonical.com/juju/docs/juju-cli/3.6/reference/juju-cli/list-of-juju-cli-commands/model-defaults/) *before* creating those models. Defaults do not change the model created above: ```shell juju model-defaults --file=./cloudinit-userdata.yaml ``` ### K8s On Kubernetes, kernel parameters are applied per **worker node**, not per container: `vm.max_map_count`, `vm.swappiness`, and `fs.file-max` are node-wide settings that the workload pods inherit from the host they are scheduled on. **Configure sysctl on each Kubernetes node** On **each node** that may run OpenSearch pods, run the following command to add the settings to a config file: ```shell sudo tee /etc/sysctl.d/opensearch.conf < ### Configure TCP retries (optional) ### VM The VM charm sets `net.ipv4.tcp_retries2` automatically; no separate configuration is needed. ### K8s The K8s charm does not set the pod-scoped `net.ipv4.tcp_retries2`. For optional `net.ipv4.tcp_retries2=5` tuning, install the [`data-platform-k8s-mutator`](https://github.com/canonical/data-platform-k8s-mutator) **before deploying OpenSearch**; it does not update existing workloads. 1. Follow the [mutator prerequisites](https://github.com/canonical/data-platform-k8s-mutator#prerequisites): check admission registration: ```shell kubectl api-versions | grep admissionregistration.k8s.io/v1 ``` Allow the unsafe sysctl on **every eligible worker node** and restart each kubelet after changing its configuration. On the Canonical Kubernetes `k8s` snap, run the following on each of those nodes: ```shell echo '--allowed-unsafe-sysctls=net.ipv4.tcp_retries2' | sudo tee -a /var/snap/k8s/common/args/kubelet sudo snap restart k8s.kubelet ``` Use your distribution’s method elsewhere. 2. With `uv`, `openssl`, and cluster-admin `kubectl` available, use the [mutator’s bootstrap script](https://github.com/canonical/data-platform-k8s-mutator#quick-start) from its own repository (not this one): ```shell git clone https://github.com/canonical/data-platform-k8s-mutator.git cd data-platform-k8s-mutator uv run python -m scripts.bootstrap_webhook --namespace webhooks \ --image ghcr.io/canonical/data-platform-k8s-mutator:1.0-24.04_edge \ --target-container-names opensearch \ --target-namespaces --dry-run ``` Replace `` with the selected Juju model’s Kubernetes namespace (usually its short name; check with `kubectl get namespaces`). `webhooks` hosts the mutator. The [image](https://github.com/canonical/data-platform-k8s-mutator/pkgs/container/data-platform-k8s-mutator) is an edge example; use a cluster-accessible image. Review the generated YAML in `deploy/`. Dry-run also writes TLS private keys there; keep them private. Then run the same command **without** `--dry-run` to deploy: ```shell uv run python -m scripts.bootstrap_webhook --namespace webhooks \ --image ghcr.io/canonical/data-platform-k8s-mutator:1.0-24.04_edge \ --target-container-names opensearch \ --target-namespaces ``` 3. Check the webhook before deploying OpenSearch: ```shell kubectl -n webhooks get deployment,pods kubectl get mutatingwebhookconfiguration sysctl-webhook ``` The webhook has one replica by default and can block controller creation cluster-wide when unavailable; review the [HA guidance](https://github.com/canonical/data-platform-k8s-mutator#high-availability-ha) before production use. ## Deploy OpenSearch For a single-host deployment, we recommend the default `testing` [profile](https://canonical.com/data/opensearch/docs/2//how-to/optimize-cluster-performance.md#how-to-optimize-cluster-performance), which sets a 1 GB JVM heap per unit. For production, meet the [profile’s resource and node-role requirements](https://canonical.com/data/opensearch/docs/2//how-to/optimize-cluster-performance.md#how-to-optimize-cluster-performance) before setting `production`: three units can cover both cluster-manager and data roles when the roles are combined, but each needs sufficient resources. Choose a command for your substrate. All examples deploy three units of an application named `opensearch`. ### VM Deploy with the default `testing` profile: ```shell juju deploy opensearch --channel=2/stable -n 3 ``` Or deploy with the `production` profile: ```shell juju deploy opensearch --channel=2/stable -n 3 --config profile=production ``` ### K8s The Kubernetes charm requires the `--trust` flag to access the model’s cloud credentials and manage Kubernetes resources such as Services and StatefulSets on your behalf. Deploy the `opensearch-k8s` charm with the default `testing` profile: ```shell juju deploy opensearch-k8s opensearch --channel=2/edge -n 3 --trust ``` To use a specific Kubernetes storage class for both OpenSearch volumes, first ensure an existing Juju `kubernetes` storage pool in this model selects that class and can provision both volumes. Find available pools with `juju storage-pools`. Pass the **pool name**, not the Kubernetes storage class, to `--storage`: ```shell juju deploy opensearch-k8s opensearch --channel=2/edge -n 3 --trust \ --storage opensearch-data=,10G \ --storage opensearch-logs=,2G ``` Or deploy with the `production` profile: ```shell juju deploy opensearch-k8s opensearch --channel=2/edge -n 3 --trust --config profile=production ``` #### NOTE The charm pulls a pinned OpenSearch workload from a [charmed-opensearch-rock](https://github.com/canonical/charmed-opensearch-rock/pkgs/container/charmed-opensearch) OCI image rather than from a snap. The image is published together with the charm revision, so no additional resource needs to be specified at deploy time. If you installed the optional [mutator](#how-to-deploy-tcp-retries), inspect a new pod: ```shell kubectl -n get pod opensearch-0 -o yaml ``` Under `securityContext.sysctls`, look for `name: net.ipv4.tcp_retries2` and `value: "5"`. If absent, check the target namespace and `kubectl -n webhooks logs deployment/sysctl-webhook`. If the `opensearch` container is running, check the effective value: ```shell kubectl -n exec opensearch-0 -c opensearch -- sysctl net.ipv4.tcp_retries2 ``` The container might not run until TLS is configured. ## Check the deployment To check the current status of the application: ```shell juju status ``` Once the units have been provisioned and the other prerequisites are met, the `opensearch` application should become `blocked` with a `Missing TLS relation with this cluster` message. If provisioning or storage is still pending, `juju status` may show a different state first. Charmed OpenSearch requires TLS encryption to start, on both the HTTP and Transport layers. ## Next steps - [Enable TLS encryption](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-enable-tls-encryption) - [Launch a large deployment](https://canonical.com/data/opensearch/docs/2//how-to/deploy/large.md#how-to-deploy-large) - [Integrate with an application](https://canonical.com/data/opensearch/docs/2//how-to/integrate-with-an-application.md#how-to-integrate-with-an-application) - [Scale horizontally](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally) - [Enable monitoring](https://canonical.com/data/opensearch/docs/2//how-to/monitoring.md#how-to-monitoring) # index.html.md # How to launch a large deployment This guide shows how to deploy a multi-application OpenSearch cluster using Juju, with dedicated node roles for scalability and fault tolerance. It covers both the **VM** charm (`opensearch`) and the **Kubernetes** charm (`opensearch-k8s`). All applications in this example are deployed in the same Juju model, using the same substrate — for Kubernetes, within a single Kubernetes cluster. The applications are integrated to form a single OpenSearch cluster. For background on node roles, data tiers, and the main orchestrator pattern, see [Node roles and cluster topology](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles). ## Prerequisites Complete the preparation steps in the [standard deployment guide](https://canonical.com/data/opensearch/docs/2//how-to/deploy/standard.md#how-to-deploy-standard) — substrate preparation, controller and model setup, kernel parameter configuration, and the optional TCP retries tuning — then return here before deploying OpenSearch. The Kubernetes charm requires the `--trust` flag to access the Kubernetes API and manage Kubernetes resources for the OpenSearch deployment. Unlike on VMs, this does not grant access to the model’s cloud credentials. ## Node roles Roles are assigned at the Juju application level (all units in an application share the same roles). If no roles are configured, the charm auto-assigns: `data`, `ingest`, `ml`, `cluster_manager`. To set roles at deploy time: ### VM ```shell juju deploy opensearch -n 3 --config roles="cluster_manager,data,ml" --channel 2/stable ``` ### K8s ```shell juju deploy opensearch-k8s opensearch -n 3 --config roles="cluster_manager,data,ml" --channel 2/edge --trust ``` To change roles after deployment (triggers a rolling restart): ```shell juju config opensearch roles="cluster_manager,data,ml" ``` #### NOTE The `cluster_manager` role cannot be removed. In a large deployment, the `data` role can be removed only if another application still holds it. See [Node roles and cluster topology](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles) for details. ## Deploy the cluster applications A large deployment consists of multiple Juju applications integrated together, each configured with specific node roles. See [Node roles and cluster topology](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles) for an explanation of the main orchestrator, failover, and data-node pattern. #### CAUTION The examples below use the `testing` profile (sets the JVM heap size to 1 GB per node), which is suited to development and testing. For production, deploy each application with `--config profile=production` (JVM heap set to 50% of RAM, capped at 31 GB) and sufficient resources. See [How to optimize cluster performance with profiles](https://canonical.com/data/opensearch/docs/2//how-to/optimize-cluster-performance.md#how-to-optimize-cluster-performance) for details on the available profiles. #### NOTE **Two critical configuration rules for large deployments:** 1. Use the same `cluster_name` value for all applications (`app` in the examples below). Applications with different `cluster_name` values cannot join the same OpenSearch cluster. See [Large deployment configuration rules](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles-large) for details, including how a cluster name is inherited when it is not explicitly configured. 2. Set `init_hold=true` on every application **except** the main orchestrator. This prevents non-orchestrator applications from starting before integration. See [Node roles and cluster topology](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles) for details on why these rules are required. ### 1. Deploy the main orchestrator Deploy the `main` application: ### VM ```shell juju deploy -n 3 \ opensearch main \ --config cluster_name="app" \ --channel 2/stable ``` ### K8s ```shell juju deploy -n 3 \ opensearch-k8s main \ --config cluster_name="app" \ --channel 2/edge \ --trust ``` Since no roles are specified, the charm auto-assigns all default roles. ### 2. Deploy a failover application (recommended) The failover application takes over orchestration if the `main` app fails or is removed: ### VM ```shell juju deploy -n 3 \ opensearch failover \ --config cluster_name="app" \ --config init_hold="true" \ --config roles="cluster_manager" \ --channel 2/stable ``` ### K8s ```shell juju deploy -n 3 \ opensearch-k8s failover \ --config cluster_name="app" \ --config init_hold="true" \ --config roles="cluster_manager" \ --channel 2/edge \ --trust ``` ### 3. Deploy data nodes Deploy an application with `data.hot` roles: ### VM ```shell juju deploy -n 3 \ opensearch data-hot \ --config cluster_name="app" \ --config roles="data.hot" \ --config init_hold="true" \ --channel 2/stable ``` ### K8s ```shell juju deploy -n 3 \ opensearch-k8s data-hot \ --config cluster_name="app" \ --config roles="data.hot" \ --config init_hold="true" \ --channel 2/edge \ --trust ``` ### 4. Deploy TLS certificates Deploy a TLS certificate provider: ```shell juju deploy self-signed-certificates ``` Track deployment progress: ```shell juju status --watch 1s ``` At this point, once the units have been provisioned, `main` will show `blocked` (missing TLS), while `failover` and `data-hot` will show `blocked` (waiting for peer cluster relation). ## Configure TLS encryption Charmed OpenSearch requires TLS. Integrate `self-signed-certificates` with all OpenSearch applications: ```shell juju integrate main:certificates self-signed-certificates juju integrate failover:certificates self-signed-certificates juju integrate data-hot:certificates self-signed-certificates ``` The `main` app will become `active` once TLS is configured. The other apps remain `blocked` until the peer-cluster relations are added in the next step. ## Form the OpenSearch cluster Integrate the applications via the peer-cluster relations to form a single OpenSearch cluster: ```shell juju integrate main:peer-cluster-orchestrator failover:peer-cluster juju integrate main:peer-cluster-orchestrator data-hot:peer-cluster juju integrate failover:peer-cluster-orchestrator data-hot:peer-cluster ``` The `main` application orchestrates cluster formation. Track progress with: ```shell juju status --watch 1s ``` Once all applications show `active`, the cluster is fully formed and operational. #### CAUTION The cluster will not come online if no `data` nodes are available. Ensure `data` nodes are deployed and ready before forming the cluster. ## Next steps - [Manage TLS encryption](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-enable-tls-encryption) — configure and rotate TLS certificates. - [Enable monitoring (COS)](https://canonical.com/data/opensearch/docs/2//how-to/monitoring.md#how-to-monitoring) — observe the cluster. - [Scale a cluster horizontally](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally) — adjust cluster size. # index.html.md # Charmed OpenSearch Dashboards [OpenSearch Dashboards](https://opensearch.org/docs/latest/dashboards/) is a frontend application that lets you visualize data stored in an OpenSearch database. Charmed OpenSearch Dashboard is the adaptation of the OpenSearch Dashboards user interface to the [Juju](https://juju.is/) environment. [OpenSearch](https://opensearch.org/) is a distributed search and analytics engine that supports various use cases, from implementing a search box on a website to analyzing security data for threat detection. #### NOTE This charm is designed to work with [Charmed OpenSearch](https://charmhub.io/opensearch). See the [Charmed OpenSearch documentation](https://canonical.com/data/opensearch/docs/). The OpenSearch Dashboards and OpenSearch operators are available for physical/virtual machines. ## In this documentation | | | |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Tutorials](https://canonical.com/data/opensearch/docs/2//dashboards/docs/tutorial/index.md#dashboards-tutorial)
Get started - a hands-on introduction to using the Charmed OpenSearch Dashboards operator for new users
| [How-to guides](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/index.md#dashboard-how-to-index)
Step-by-step guides covering key operations and common tasks | ## Project and community Charmed OpenSearch Dashboards is an open source project that warmly welcomes community contributions, suggestions, fixes, and constructive feedback. * Check our [Code of Conduct](https://ubuntu.com/community/ethos/code-of-conduct) * Raise software issues or feature requests in [GitHub](https://github.com/canonical/opensearch-dashboards-operator/issues) * Report security issues through [LaunchPad](https://wiki.ubuntu.com/DebuggingSecurity#How_to_File). * Meet the community and chat with us on [Matrix](https://matrix.to/#/#charmhub-data-platform:ubuntu.com) ## Licensing & trademark The Charmed OpenSearch Dashboards Operator is free software, distributed under the [Apache Software License, version 2.0](https://github.com/canonical/opensearch-dashboards-operator/blob/main/LICENSE). All of the software in the OpenSearch project is released under the [Apache Software License, version 2.0](https://github.com/canonical/opensearch-dashboards-operator/blob/main/LICENSE) # index.html.md # How to access OpenSearch Dashboards using OAuth This guide shows how to configure OpenSearch Dashboards to support single sign-on (SSO) using the [Canonical Identity Platform](https://canonical-identity.readthedocs-hosted.com/). The platform is a Charm bundle that includes an identity provider (Hydra), an identity broker (Kratos), an ingress (Traefik), and a login-ui application. By the end, you will be able to sign in to OpenSearch Dashboards with an admin user created in Kratos. ## Prerequisites * A deployed charmed OpenSearch cluster on LXD. * A deployed charmed OpenSearch Dashboards on LXD and integrated with OpenSearch. See: [Deploy, connect, and scale](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/deploy-connect-scale.md#dashboard-how-to-deploy-connect-scale). * A deployed Canonical Identity Platform on Kubernetes. * Working Integration between OpenSearch and Canonical Identity Platform through certificates and Hydra OAuth interface. See: [How to access OpenSearch using OAuth](https://canonical.com/data/opensearch/docs/2/how-to/access-using-oauth/). #### NOTE This guide assumes that LXD and MicroK8s are managed under the same Juju controller (`overlord`, as set up in the [Tutorial](https://canonical.com/data/opensearch/docs/2//dashboards/docs/tutorial/index.md#dashboards-tutorial)). Running LXD and MicroK8s under separate controllers may cause failures during cross-model integration. If you have not yet added MicroK8s to your controller, add it as follows: ```shell # Export microk8s config microk8s.kubectl config view --raw > ~/.kube/config # Replace localhost IP by the computer public IP export LOCAL_IP="127.0.0.1" export PUBLIC_IP=$(ip -4 -j route get 2.2.2.2 | jq -r '.[] | .prefsrc') sed -i 's/'${LOCAL_IP}'/'${PUBLIC_IP}'/g' ~/.kube/config # Add the microk8s cloud to the existing controller juju add-k8s microk8s-cluster --controller overlord ``` After adding the cloud, create a model on it for the Identity Platform: ```shell juju add-model oauth microk8s-cluster ``` ## Deploy OpenSearch Dashboards On the LXD model where OpenSearch is deployed, deploy OpenSearch Dashboards, and integrate it with OpenSearch charm. ```shell juju switch overlord:tutorial juju deploy opensearch-dashboards --channel=2/edge juju integrate opensearch opensearch-dashboards ``` Now, wait for the OpenSearch and OpenSearch Dashboards to become active and ready: ```shell watch juju status ``` ## Integrate OpenSearch Dashboards with Canonical Identity Platform Switch to the `oauth` model and verify the identity platform bundle is ready: ```shell juju switch overlord:oauth juju status ```
Output example ```text Model Controller Cloud/Region Version SLA Timestamp oauth overlord microk8s-cluster/localhost 3.6.10 unsupported 15:38:54Z App Version Status Scale Charm Channel Rev Address Exposed Message hydra v2.3.0 active 1 hydra latest/edge 339 10.152.183.124 no identity-platform-login-ui-operator 0.21.2 active 1 identity-platform-login-ui-operator latest/edge 146 10.152.183.25 no kratos v1.3.1 active 1 kratos latest/edge 500 10.152.183.20 no kratos-external-idp-integrator blocked 1 kratos-external-idp-integrator latest/edge 245 10.152.183.113 no Invalid configuration: Missing required configuration 'client_id' for provider 'generic' postgresql-k8s 14.15 active 1 postgresql-k8s 14/stable 495 10.152.183.109 no self-signed-certificates active 1 self-signed-certificates 1/stable 317 10.152.183.250 no traefik-admin v2.11.0 active 1 traefik-k8s latest/stable 176 10.241.7.40 no traefik-public v2.11.0 active 1 traefik-k8s latest/stable 176 10.241.7.39 no Unit Workload Agent Address Ports Message hydra/0* active idle 10.1.156.80 identity-platform-login-ui-operator/0* active idle 10.1.156.81 kratos-external-idp-integrator/0* blocked idle 10.1.156.82 Invalid configuration: Missing required configuration 'client_id' for provider 'generic' kratos/0* active idle 10.1.156.91 postgresql-k8s/0* active idle 10.1.156.89 Primary self-signed-certificates/0* active idle 10.1.156.83 traefik-admin/0* active idle 10.1.156.90 traefik-public/0* active idle 10.1.156.86 ```
All the components of the bundle must be active except `kratos-external-idp-integrator`. It is in blocked status. Before switching back to the LXD model, offer the `hydra:oauth` and `self-signed-certificates:certificates` endpoints from the OAuth model so they can be consumed cross-model by the OpenSearch Dashboards model: ```shell juju offer hydra:oauth juju offer self-signed-certificates:certificates ``` Switch back to the OpenSearch Dashboards model and consume the offers: ```shell juju switch overlord:tutorial juju consume admin/oauth.hydra juju consume admin/oauth.self-signed-certificates ``` Now integrate OpenSearch Dashboards with the consumed offers: ```shell juju integrate opensearch-dashboards:certificates self-signed-certificates:certificates juju integrate opensearch-dashboards:oauth hydra:oauth ``` ## Create an admin account Create an admin account using Kratos. This command requires an email and username, and prints the password reset link as well as the reset code: ```shell juju run kratos/leader create-admin-account email=myuser@example.com username=myuser --model overlord:oauth ``` The output is similar to the following: ```text Running operation 1 with 1 task - task 2 on unit-kratos-0 Waiting for task 2... 15:39:52 Creating admin account. 15:39:53 Successfully created admin account: 3b322477-6250-4606-b820-b32d679b3cff. 15:39:53 Creating recovery code for resetting admin password. expires-at: "2025-09-25T16:39:52.595790816Z" identity-id: 3b322477-6250-4606-b820-b32d679b3cff password-reset-code: "868748" password-reset-link: https://10.241.7.39/welcome-k8s-identity-platform-login-ui-operator/ui/reset_email?flow=a5966798-646b-46e7-ae6f-a466e30323a9 ``` The output provides a password reset link and recovery code. Open the link, enter the recovery code, and set a password. Make sure to enter the recovery code given in the output of the previous command. Once that is done, you are redirected to the password reset page, where you specify the user’s password. Once the password is set, you are prompted to configure 2FA (mandatory). ## Access OpenSearch Dashboards using Single Sign-On To access OpenSearch Dashboards, use the IP address on the `opensearch-dashboards/0` unit to form the URL: `https://:5601`. Once the account is ready, open OpenSearch Dashboards. A **Log in with single sign-on** button appears. ![image](dashboards/docs/how-to/img/OSD-OAuth-1.jpg) Click the button to open the identity platform login UI. You are redirected to the identity platform UI login screen, where you are prompted to enter the email and password. ![image](dashboards/docs/how-to/img/OSD-OAuth-2.jpg) If it is your first time connecting, it also asks for the 2FA code. ![image](dashboards/docs/how-to/img/OSD-OAuth-3.jpg) After a successful login, you are redirected to the OpenSearch Dashboards home screen. ![image](dashboards/docs/how-to/img/OSD-OAuth-4.jpg) ## Next steps * Review the *roles mapping* section in [*How to access OpenSearch using OAuth*](https://canonical.com/data/opensearch/docs/2/how-to/access-using-oauth/) to assign permissions. * Follow the guide [How to manage external identity providers](https://canonical-identity.readthedocs-hosted.com/how-to/manage-external-identity-providers/) to enable logins with providers like GitHub. # index.html.md # How-to guides The following guides cover key processes and common tasks for Charmed OpenSearch Dashboards. Initial setup, including: [deploying, connecting, and scaling](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/deploy-connect-scale.md#dashboard-how-to-deploy-connect-scale), [managing security](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/manage-security.md#dashboard-how-to-manage-security), [enabling JWT Auth](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/enable-jwt-authentication.md#dashboard-how-to-enable-jwt-auth), and [using OAuth](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/access-using-oauth.md#dashboard-how-to-access-using-oauth). Configuration and maintenance: [monitoring](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/enable-monitoring.md#dashboard-how-to-enable-monitoring). # index.html.md # How to enable JWT authentication This document shows how to enable authentication with JSON Web Tokens (JWT) in OpenSearch. ## Prerequisites * A running deployment of OpenSearch on VM with minimum charm revision 275 (22.04) or 276 (24.04) * A valid JSON Web Token for testing, issued by the JWT provider of your choice * The signing-key with which the JWT was signed The generation of JWTs is not part of the scope of this document. ## Deploy and configure JWT integrator The configuration for JWT authentication is provided to OpenSearch via the JWT integrator charm. Deploy the charm: ```shell juju deploy jwt-integrator --channel 1/edge ``` After the deployment has settled, the charm is in blocked status because of missing configuration. Check this with `juju status`: ```text opensearch dev-controller localhost/localhost 3.6.8 unsupported 12:02:14Z App Version Status Scale Charm Channel Rev Exposed Message jwt-integrator blocked 1 jwt-integrator 1/edge 1 no Missing 'signing-key' or 'roles-key' con... Run `status-detail`: 0 action required; 1 additional statuses. opensearch active 3 opensearch 2/edge 276 no self-signed-certificates active 1 self-signed-certificates 1/stable 317 no Unit Workload Agent Machine Public address Ports Message jwt-integrator/0* blocked idle 3 10.65.50.163 Missing 'signing-key' or 'roles-key' con... Run `status-detail`: 0 action required; 1 additional statuses. opensearch/0* active idle 0 10.65.50.55 9200/tcp opensearch/1 active idle 1 10.65.50.63 9200/tcp opensearch/2 active idle 2 10.65.50.37 9200/tcp self-signed-certificates/0* active idle 4 10.65.50.79 ``` Now, configure the JWT parameters to JWT integrator. First, create a secret containing the signing-key: ```shell juju add-secret jwt-key signing-key="" ``` Take a note of the secret URI for later. Now grant permissions for the secret to JWT integrator: ```shell juju grant-secret jwt-key jwt-integrator ``` Next step is to provide the secret URI as configuration option: ```shell juju config jwt-integrator signing-key= ``` Configure all other configuration options, according to your JWT provider: ```shell juju config jwt-integrator roles-key=role subject-key=user jwt-url-parameter=jwt ... ``` ## Enable JWT in OpenSearch Now it’s time to enable JWT authentication in OpenSearch. This is done by integrating the JWT integrator with OpenSearch: ```shell juju integrate jwt-integrator opensearch ``` After a few moments, OpenSearch has applied the provided configuration and updated its security plugin. Now query OpenSearch with your JWT: ```shell curl --header "Authorization: Bearer " -XGET "https://:9200/_cat/nodes" -k ``` ## Large deployments If you have an OpenSearch large deployments cluster, it is required to integrate the JWT integrator with the main-orchestrator of your large deployment. Check which of your deployed OpenSearch applications is the main-orchestrator by checking the integrations section of `juju status --relations`: ```text Integration provider Requirer Interface Type Message opensearch-main:peer-cluster-orchestrator opensearch-data:peer-cluster peer_cluster regular ``` Now integrate the JWT integrator with this application: ```shell juju integrate jwt-integrator opensearch-main ``` If the wrong application of your large deployments is integrated with the JWT integrator, you will see a `blocked` status: ```text App Version Status Scale Charm Channel Rev Exposed Message jwt-integrator active 1 jwt-integrator 1/edge 1 no opensearch-data blocked 3 opensearch 3 no JWT relation must be created with Main-cluster-orchestrator opensearch-failover active 1 opensearch 1 no opensearch-main active 1 opensearch 2 no self-signed-certificates active 1 self-signed-certificates 1/stable 317 no ``` ## Enabling JWT authentication with OpenSearch Dashboards To use JWT authentication in the OpenSearch Dashboards UI, all you need to do is integrate the JWT integrator with OpenSearch Dashboards: ```shell juju integrate jwt-integrator opensearch-dashboards ``` Make sure you have configured the configuration option `jwt-url-parameter` to the JWT integrator: ```shell juju config jwt-integrator jwt-url-parameter="jwt" ``` After a few moments, access it by adding your JWT as a URL parameter in a web browser: ```shell http://:5601?jwt= ``` # index.html.md # How to enable monitoring (COS) #### NOTE All commands are written for Juju 3.4 or later. The Kubernetes charm forwards its workload logs with Pebble log forwarding, which requires Juju 3.4 or later. The Canonical Observability Stack (COS) collects metrics and logs from Charmed OpenSearch Dashboards. It provides a Grafana dashboard, Prometheus alert rules, and logs in Loki. The integration depends on where Dashboards runs. On VMs, `opensearch-dashboards` exposes the `cos-agent` endpoint to a machine `grafana-agent`, which forwards telemetry to COS Lite. On Kubernetes, `opensearch-dashboards-k8s` exposes `metrics-endpoint`, `grafana-dashboard`, and `logging`, and a `grafana-agent-k8s` charm in the Dashboards model collects and forwards the telemetry to COS Lite. If COS Lite runs in the same model, Dashboards can integrate with the COS applications directly instead. ## Prerequisites ### VM * A deployed [Charmed OpenSearch Dashboards operator](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/deploy-connect-scale.md#dashboard-how-to-deploy-connect-scale) in a machine model, integrated with Charmed OpenSearch. * A deployed [COS Lite bundle on Kubernetes](https://documentation.ubuntu.com/observability/track-3.0/tutorial/cos-lite-microk8s-sandbox/) in a separate model. If using a VM and MicroK8s under the same Juju controller, see the [OAuth guide](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/access-using-oauth.md#dashboard-how-to-access-using-oauth) for how to add MicroK8s to that controller. Create a separate `cos` model on it with `juju add-model cos microk8s-cluster` before deploying COS Lite. ### K8s * A deployed [`opensearch-dashboards-k8s`](https://charmhub.io/opensearch-dashboards-k8s) application in a Kubernetes model, integrated with Charmed OpenSearch. * A deployed [COS Lite bundle on Kubernetes](https://documentation.ubuntu.com/observability/track-3.0/tutorial/cos-lite-microk8s-sandbox/), either in a separate model or alongside Dashboards in the same model. COS [recommends a separate model](https://documentation.ubuntu.com/observability/track-3.0/reference/topology/#deploy-in-isolation). If COS Lite and Dashboards run in the same model, skip the next two sections and [integrate directly](#dashboard-integrate-with-cos). The steps below use `:` for the COS Lite model and `:` for the Dashboards model. For example, if both models are on the `overlord` controller, use `overlord:cos` and `overlord:tutorial`. Substitute your own controller, model, and offer owner as needed. ## Offer COS interfaces If COS Lite runs in a separate model from Dashboards, switch to the COS model and offer its endpoints: ```shell juju switch : juju offer grafana:grafana-dashboard grafana-dashboards juju offer loki:logging loki-logging juju offer prometheus:receive-remote-write prometheus-receive-remote-write ``` If you deployed COS Lite with the [offers overlay](https://github.com/canonical/cos-lite-bundle/blob/main/overlays/offers-overlay.yaml), these offers already exist; do not create them again. ## Consume offers If COS Lite runs in a separate model from Dashboards, switch to the Dashboards model and consume the offers: ```shell juju switch : juju consume :admin/.grafana-dashboards juju consume :admin/.loki-logging juju consume :admin/.prometheus-receive-remote-write ``` ## Integrate with COS ### VM Deploy [grafana-agent](https://charmhub.io/grafana-agent) in the Dashboards model: ```shell juju deploy grafana-agent ``` Integrate it with the consumed COS offers: ```shell juju integrate grafana-agent grafana-dashboards juju integrate grafana-agent loki-logging juju integrate grafana-agent prometheus-receive-remote-write ``` Then integrate it with Dashboards: ```shell juju integrate grafana-agent opensearch-dashboards:cos-agent ``` ### K8s On Kubernetes, `opensearch-dashboards-k8s` stays blocked with the message `Ingress relation missing` until it has an ingress relation. If the model already has an ingress, integrate Dashboards with the existing application. Otherwise, deploy [`traefik-k8s`](https://charmhub.io/traefik-k8s) and integrate it with Dashboards: ```shell juju deploy traefik-k8s --trust juju integrate opensearch-dashboards-k8s:ingress traefik-k8s:ingress ``` For more about ingress with the Traefik charm, see the [Traefik operator documentation](https://canonical.com/juju/docs/traefik-k8s-charm/latest/). #### NOTE If `grafana-agent-k8s` is already deployed in this model (for example, while monitoring Charmed OpenSearch), skip this step (agent deployment) and the next step (COS offers integration); only add the Dashboards integrations below. Deploy [grafana-agent-k8s](https://charmhub.io/grafana-agent-k8s) in the Dashboards model: ```shell juju deploy grafana-agent-k8s --trust ``` Integrate it with the consumed COS offers: ```shell juju integrate grafana-agent-k8s:grafana-dashboards-provider grafana-dashboards juju integrate grafana-agent-k8s:logging-consumer loki-logging juju integrate grafana-agent-k8s:send-remote-write prometheus-receive-remote-write ``` Then integrate it with Dashboards: ```shell juju integrate opensearch-dashboards-k8s:metrics-endpoint grafana-agent-k8s:metrics-endpoint juju integrate opensearch-dashboards-k8s:grafana-dashboard grafana-agent-k8s:grafana-dashboards-consumer juju integrate opensearch-dashboards-k8s:logging grafana-agent-k8s:logging-provider ``` * `metrics-endpoint` lets the agent scrape the Dashboards metrics endpoint. * `grafana-dashboard` transfers the **Charmed Opensearch Dashboards** dashboard. * `logging` sends the Dashboards logs. If COS Lite and Dashboards run in the same model, skip the agent and the offer and consume sections above. Integrate Dashboards with the COS applications directly: ```shell juju integrate opensearch-dashboards-k8s:metrics-endpoint prometheus:metrics-endpoint juju integrate opensearch-dashboards-k8s:grafana-dashboard grafana:grafana-dashboard juju integrate opensearch-dashboards-k8s:logging loki:logging ``` After integration, Grafana displays the **Charmed Opensearch Dashboards** dashboard (with the charm revision appended, for example `Charmed Opensearch Dashboards - Rev 10`), and Loki receives Dashboards logs. ## Connect to Grafana To connect to the Grafana web interface, follow the [Browse dashboards](https://documentation.ubuntu.com/observability/track-3.0/tutorial/cos-lite-microk8s-sandbox/#browse-dashboards) section of the MicroK8s “Getting started” guide. Obtain the admin password from the model where COS Lite runs (the Dashboards model if you used the same-model K8s option): ```shell juju run grafana/leader get-admin-password --model : ``` For details on available metrics and default alert rules, see the [Monitoring reference: metrics and alert rules](https://canonical.com/data/opensearch/docs/2//dashboards/docs/reference/monitoring.md#dashboard-reference-monitoring). ## Logs To view OpenSearch Dashboards logs, open `Home > Explore` in Grafana. In `Label filters`, set `juju_application` to your Dashboards application name (for example, `opensearch-dashboards` on VMs or `opensearch-dashboards-k8s` on Kubernetes). Select an operation such as `Line contains` and run the query. The following screenshot shows a VM application as an example: ![image|690x313](dashboards/docs/how-to/img/OSD-Monitoring-img1.png) # index.html.md # How to deploy, connect, and scale This guide covers deploying the OpenSearch Dashboards charm, connecting it to an OpenSearch cluster, and scaling the number of units up or down. ## Prerequisites A Juju model containing: * An active `opensearch` application * A TLS certificate provider charm (e.g. `self-signed-certificates`) integrated with `opensearch` To learn how to set up and deploy an OpenSearch application, see steps 1, 2, and 3 of the [OpenSearch Tutorial](https://canonical.com/data/opensearch/docs/2/tutorial/1-set-up-the-environment/). Make sure you’ve set up the correct [kernel parameters](https://canonical.com/data/opensearch/docs/2/tutorial/1-set-up-the-environment/#set-kernel-parameters) for your OpenSearch deployment. For a detailed walk-through of a full deployment on LXD, see the [Tutorial](https://canonical.com/data/opensearch/docs/2//dashboards/docs/tutorial/index.md#dashboards-tutorial). ## Deploy OpenSearch Dashboards On top of a live, healthy OpenSearch database, deploy the Dashboards visualization interface: ```shell juju deploy opensearch-dashboards --channel=2/edge ``` ## Connect to OpenSearch Integrate the Dashboards charm with the OpenSearch database: ```shell juju integrate opensearch opensearch-dashboards ``` As a result, a healthy system should look something like this: ```text Model Controller Cloud/Region Version SLA Timestamp tutorial overlord localhost/localhost 3.5.3 unsupported 17:40:00+02:00 App Version Status Scale Charm Channel Rev Exposed Message opensearch active 2 opensearch 2/edge 159 no opensearch-dashboards active 1 opensearch-dashboards 2/edge 20 no self-signed-certificates active 1 self-signed-certificates 1/stable 317 no Unit Workload Agent Machine Public address Ports Message opensearch-dashboards/0* active idle 3 10.34.169.173 5601/tcp opensearch/0 active idle 0 10.34.169.84 9200/tcp opensearch/1* active idle 1 10.34.169.242 9200/tcp self-signed-certificates/0* active idle 2 10.34.169.5 Machine State Address Inst id Base AZ Message 0 started 10.34.169.84 juju-df6483-0 ubuntu@22.04 Running 1 started 10.34.169.242 juju-df6483-1 ubuntu@22.04 Running 2 started 10.34.169.5 juju-df6483-2 ubuntu@22.04 Running 3 started 10.34.169.173 juju-df6483-3 ubuntu@22.04 Running Integration provider Requirer Interface Type Message opensearch-dashboards:dashboard_peers opensearch-dashboards:dashboard_peers dashboard_peers peer opensearch-dashboards:restart opensearch-dashboards:restart rolling_op peer opensearch-dashboards:upgrade opensearch-dashboards:upgrade upgrade peer opensearch:node-lock-fallback opensearch:node-lock-fallback node_lock_fallback peer opensearch:opensearch-client opensearch-dashboards:opensearch-client opensearch_client regular opensearch:opensearch-peers opensearch:opensearch-peers opensearch_peers peer opensearch:upgrade-version-a opensearch:upgrade-version-a upgrade peer self-signed-certificates:certificates opensearch-dashboards:certificates tls-certificates regular self-signed-certificates:certificates opensearch:certificates tls-certificates regular ``` ## Scale up/down It’s very easy to increase or decrease the number of units in a Juju system. Scaling up goes as: ```shell juju add-unit opensearch-dashboards -n ``` While scaling down goes as: ```shell juju remove-unit opensearch-dashboards/ ``` # index.html.md # How to manage security This guide covers enabling TLS encryption and changing the `kibanaserver` credentials for Charmed OpenSearch Dashboards. ## Enable TLS encryption First, make sure you have a TLS certificates charm set up. This guide shows how to enable TLS using the [`self-signed-certificates` charm](https://github.com/canonical/self-signed-certificates-operator) as an example. #### CAUTION **[Self-signed certificates](https://en.wikipedia.org/wiki/Self-signed_certificate) are not recommended for a production environment.** Check the [X.509 certificates topic](https://charmhub.io/topics/security-with-x-509-certificates) for an overview of the signed and self-signed certificate charms available. To deploy the `self-signed-certificates` charm: ```shell juju deploy self-signed-certificates --config ca-common-name="Tutorial CA" ``` Then, integrate it with the OpenSearch Dashboards charm: ```shell juju integrate self-signed-certificates opensearch-dashboards ``` For further guidance on how to manage TLS certificates, see the [TLS encryption page for the OpenSearch charm](https://canonical.com/data/opensearch/docs/2/how-to/tls-encryption/). ## Change credentials Dashboards have a “super-user” called `kibanaserver`, that is a built-in user set in the OpenSearch database. For this reason, the credentials change doesn’t happen on the Dashboards side, rather on the OpenSearch side. Running the following command on the leader unit changes the `kibanaserver` password: ```shell juju run opensearch/leader set-password username=kibanaserver ``` The `set-password` action defaults to the `admin` user, so the `username` parameter is required to target `kibanaserver`. The new credentials are populated for the Dashboards charm. # index.html.md # Reference * [Monitoring reference: metrics and alert rules](https://canonical.com/data/opensearch/docs/2//dashboards/docs/reference/monitoring.md) * [Release notes](https://canonical.com/data/opensearch/docs/2//dashboards/docs/reference/release-notes/index.md) * [Revision 21](https://canonical.com/data/opensearch/docs/2//dashboards/docs/reference/release-notes/revision-21.md) # index.html.md # Monitoring reference: metrics and alert rules This reference describes the OpenSearch Dashboards metrics and alert rules available through COS on VMs and Kubernetes. To set up COS, see [How to enable monitoring](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/enable-monitoring.md#dashboard-how-to-enable-monitoring). ## Metrics Both the OpenSearch Dashboards snap on VMs and the Charmed OpenSearch Dashboards rock on Kubernetes run [the Prometheus Exporter for OpenSearch Dashboards](https://github.com/canonical/prometheus-opensearch-dashboards-exporter), which serves metrics on port 9684 at `/metrics`. The machine charm shares its metrics with COS through `cos-agent`. On Kubernetes, `opensearch-dashboards-k8s` provides a `metrics-endpoint` for Prometheus scraping, either directly or through `grafana-agent-k8s`. For descriptions of the exporter metrics, see its [README](https://github.com/canonical/prometheus-opensearch-dashboards-exporter?tab=readme-ov-file#metrics). ## Default alert rules To ensure you are referencing the latest default alert rules, check the source file for your charm variant: * VM: [machine Prometheus alert rules](https://github.com/canonical/opensearch-dashboards-operator/blob/2/edge/machine/src/alert_rules/prometheus/prometheus_alerts.yaml) * K8s: [Kubernetes Prometheus alert rules](https://github.com/canonical/opensearch-dashboards-operator/blob/2/edge/kubernetes/src/alert_rules/prometheus/prometheus_alerts.yaml) The Kubernetes charm also ships [Loki log-based alert rules](https://github.com/canonical/opensearch-dashboards-operator/blob/2/edge/kubernetes/src/loki_alert_rules/dashboards_logs.rule). Their checked-in expressions filter on `juju_application="opensearch-dashboards"`; they will not match the default `opensearch-dashboards-k8s` application name without changing that filter. Both charms include these Prometheus alert rules: | Alert | Severity | Notes | |--------------------------------------|---------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | OpenSearchDashboardsScrapeFailed | ![critical](https://img.shields.io/badge/critical-red) | Triggered when the prometheus scrape fails. | | OpenSearchDashboardsRed | ![critical](https://img.shields.io/badge/critical-red) | OpenSearch Dashboards status can turn red for the following reasons:

- Incompatibility between OpenSearch Dashboards and OpenSearch Service
- Insufficient memory
- OpenSearch is in red state | | OpenSearchDashboardsYellow | ![warning](https://img.shields.io/badge/warning-yellow) | Triggered when OpenSearch Dashboards is yellow. Dashboard plugins might be degraded or shards may be relocating or initializing. | | OpenSearchDashboardsPluginRed | ![critical](https://img.shields.io/badge/critical-red) | Triggered when OpenSearch Dashboards plugin or core component are in red state. | | OpenSearchDashboardsPluginYellow | ![warning](https://img.shields.io/badge/warning-yellow) | Triggered when OpenSearch Dashboards plugin or core component are in yellow state. | | OpenSearchDashboardsNoMetrics | ![critical](https://img.shields.io/badge/critical-red) | Triggered when exporter failed to collect status metrics. | | OpenSearchDashboardsLongResponseTime | ![critical](https://img.shields.io/badge/critical-red) | Triggered when the server is up and responsive, however with a high latency. | # index.html.md # Release notes * [Revision 21](https://canonical.com/data/opensearch/docs/2//dashboards/docs/reference/release-notes/revision-21.md) # index.html.md # Revision 21 24 September 2024 Charmed OpenSearch Dashboards Revision 21 has been deployed to the [`2/stable` channel](https://charmhub.io/opensearch-dashboards?channel=2/stable) on Charmhub. ## Requirements and compatibility * Juju `v3.5.3+` * Older minor versions of Juju 3 may be compatible, but are not officially supported. * This charm was built to be integrated with [Charmed OpenSearch](https://charmhub.io/opensearch) * See: [Deploy, connect, and scale](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/deploy-connect-scale.md#dashboard-how-to-deploy-connect-scale) ## Highlights * Encryption via TLS * See: [Manage security](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/manage-security.md#dashboard-how-to-manage-security) * High availability deployment support * Automated rolling restart * Monitoring via COS * See: [Enable monitoring](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/enable-monitoring.md#dashboard-how-to-enable-monitoring) ## Join the community Charmed OpenSearch Dashboards is an official distribution of [OpenSearch Dashboards](https://opensearch.org/docs/latest/dashboards/). It’s an open-source project that welcomes community contributions, suggestions, fixes and constructive feedback. * Raise an issue or feature request in the [Github repository](https://github.com/canonical/opensearch-dashboards-operator/issues). * Meet the community and chat with us in our [Matrix channel](https://matrix.to/#/#charmhub-data-platform:ubuntu.com) * See the Charmed OpenSearch [contribution guidelines](https://github.com/canonical/opensearch-operator/blob/main/CONTRIBUTING.md) on GitHub and read the Ubuntu Community’s [Code of Conduct](https://ubuntu.com/community/code-of-conduct). # index.html.md # Tutorial This tutorial guides you through how to get Charmed OpenSearch Dashboards up and running. It also provides a quick introduction to the following: * Set up an environment using [Multipass](https://multipass.run/) with [LXD](https://ubuntu.com/lxd) and [Juju](https://juju.is/). * Deploy OpenSearch Dashboards using a single command. * Configure TLS certificate equally simple. This tutorial focuses on OpenSearch Dashboards deployment, assuming that you have familiarity with: * Basic terminal commands. * OpenSearch and OpenSearch Dashboards concepts * The [Charmed OpenSearch Operator](https://charmhub.io/opensearch) ## Set up the environment In this section, you will set up your environment by: * installing and setting up Multipass * installing Juju and bootstrapping LXD * setting up a graphical interface with Multipass ### Install and set up Multipass [Multipass](https://multipass.run/) is a quick and easy way to launch virtual machines running Ubuntu. It uses the “[cloud-init](https://cloud-init.io/)” standard to install and configure all the necessary parts automatically. Installation of Multipass from [Snap](https://snapcraft.io/multipass) and launching a new VM using “[charm-dev](https://github.com/canonical/multipass-blueprints/blob/main/v1/charm-dev.yaml)” cloud-init config goes as: ```shell sudo snap install multipass && \ multipass launch --cpus 4 --memory 8G --disk 30G --name my-vm charm-dev # tune CPU/RAM/HDD accordingly to your needs ``` The full set of launch parameters is described in the [launch command reference](https://multipass.run/docs/launch-command). Multipass [commands](https://multipass.run/docs/multipass-cli-commands) are generally short and intuitive. For example, to show all running VMs: ```shell multipass list ``` When the new VM is up and running, connect using: ```shell multipass shell my-vm ``` You can exit the Multipass VM using `Ctrl + D` or the exit command. ### Install and set up Juju The next step is to install Juju and initialize [LXD](https://canonical.com/lxd) (a lightweight container hypervisor). ```shell sudo snap install juju --classic --channel=3.5/stable sudo snap install lxd lxd init --auto ``` Files `/var/log/cloud-init.log` and `/var/log/cloud-init-output.log` contain all low-level installation details. Now that LXD and Juju are installed, the next step is to bootstrap Juju to use local LXD: ```shell juju bootstrap localhost overlord ``` The controller can work with different models. Most applications such as OpenSearch or OpenSearch Dashboards. To set up a new “model” called tutorial, run: ```shell juju add-model tutorial ``` You can now view the model you created above by entering the command `juju status`. You should see something similar to the following output: ```text Model Controller Cloud/Region Version SLA Timestamp tutorial overlord localhost/localhost 3.5.3 unsupported 14:22:02+02:00 Model "admin/tutorial" is empty. ``` ### Set up a graphical interface There are graphical interfaces available for Multipass (see more details in the [Multipass Graphical Interface chapter](https://multipass.run/docs/set-up-a-graphical-interface)). We recommend to use RDP: ```shell sudo apt install ubuntu-desktop xrdp remmina-plugin-rdp remmina sudo passwd ubuntu # Set password here ``` Now you should be able to connect using the IP of the earlier Multipass list command: ```shell remmina -c rdp:// ``` If the environment comes up with a small resolution, use this great [Stack Overflow suggestion](https://askubuntu.com/questions/914775/remmina-scale-resolution-when-connect-from-ubuntu-to-windows-10). Note that after the graphical setup you may be instructed to restart the Multipass instance. You probably want to do this before installing the services within (as some may require re-initialization after a reboot otherwise). ## Get OpenSearch Dashboards up and running The objective of OpenSearch Dashboard is to display the contents of an OpenSearch database. This is why a functional OpenSearch database is a pre-requisite for the Dashboards application to install successfully. So, before going further, let’s set up Charmed OpenSearch. Note that OpenSearch has a mandatory requirement of TLS support, so we need to deploy it alongside the `self-signed-certificates` charm and integrate (also known as “relate”) them. Make sure that the environment is ready and the Juju model is correctly configured following instructions in OpenSearch Documentation [Set kernel parameters](https://canonical.com/data/opensearch/docs/2/tutorial/1-set-up-the-environment/#set-kernel-parameters). Subsequently, we can deploy OpenSearch with TLS: ```shell juju deploy opensearch --channel=2/edge -n 2 juju deploy self-signed-certificates --channel=1/stable juju integrate self-signed-certificates opensearch ``` We can simply add the OpenSearch Dashboards charm to this setup by deploying and relating it to OpenSearch ```shell juju deploy opensearch-dashboards --channel=2/edge juju integrate opensearch opensearch-dashboards ``` And there we go! Now if you check the status of your services with `juju status`. Your output should be similar to the example below: ```text Model Controller Cloud/Region Version SLA Timestamp tutorial overlord localhost/localhost 3.5.3 unsupported 16:50:56+02:00 App Version Status Scale Charm Channel Rev Exposed Message opensearch active 2 opensearch 2/edge 159 no opensearch-dashboards active 1 opensearch-dashboards 2/edge 20 no self-signed-certificates active 1 self-signed-certificates 1/stable 317 no Unit Workload Agent Machine Public address Ports Message opensearch-dashboards/0* active idle 3 10.34.169.173 5601/tcp opensearch/0 active idle 0 10.34.169.84 9200/tcp opensearch/1* active idle 1 10.34.169.242 9200/tcp self-signed-certificates/0* active idle 2 10.34.169.5 Machine State Address Inst id Base AZ Message 0 started 10.34.169.84 juju-df6483-0 ubuntu@22.04 Running 1 started 10.34.169.242 juju-df6483-1 ubuntu@22.04 Running 2 started 10.34.169.5 juju-df6483-2 ubuntu@22.04 Running 3 started 10.34.169.173 juju-df6483-3 ubuntu@22.04 Running ``` To verify the integrations, add the flag `--relations`. ```shell juju status --relations ``` Alternatively, if you want to monitor your system (with a view updating every second): ```shell watch juju status ``` ## Enable TLS encryption Charmed OpenSearch Dashboards supports HTTPS connections. Configuration is similar to what we have seen for OpenSearch – we just need to integrate the Dashboards charm against the TLS charm: ```shell juju integrate self-signed-certificates opensearch-dashboards ``` Once the two charms are successfully related, you should be able to access the same URL now using HTTPS. ## Access OpenSearch Dashboards Assuming that you have a virtual environment available (as described in [Set up the environment](#dashboards-setup-environment)), open a browser and type there the following URL: ```text http://:5601 ``` The address of the unit is available in the `juju status` output of the opensearch-dashboards unit. For example, in the output from [Get OpenSearch Dashboards up and running](#dashboards-deploy), this would be `10.163.9.173`. You should see something like this: ![OpenSearch Dashboards - Multipass Desktop](dashboards/docs/tutorial/img/multipass-desktop.jpeg) ### Set up an OpenSearch user Set up a user using the `data-integrator` [charm](https://charmhub.io/data-integrator). At deployment time the OpenSearch index (in the example: `index_name`) has to be specified as well. This is an arbitrary, alphanumerical identifier of the users’ data space in OpenSearch. User creation takes effect when integrated against the `opensearch` charm. ```bash juju deploy data-integrator --config index-name= juju integrate data-integrator opensearch ``` #### NOTE This user will have normal privileges – meaning this user will only have access to the index it owns. In case a broader access to the cluster’s indices is needed, it is possible to create an admin / privileged user as follows: ```bash juju deploy data-integrator admin --config index-name=admin-index --config extra-user-roles="admin" ``` #### CAUTION Please only create admin users when extremely needed, and handle with special care as authenticating with an admin user grants full access to all indices in the cluster. Retrieve user credentials running ```bash juju run data-integrator/0 get-credentials ``` at the bottom of the output you should see something like: ```bash password: 8kubD7nbWYZFHPVEzIVmTyqV42I7wHb4 username: opensearch-client_15 ``` ### Create the “index pattern” Log in to the Dashboard using these credentials. Clicking the top left icon the main menu will pull down. Select **Management** / **Dashboards Management** here ![OpenSearch Dashboards - Initial view](dashboards/docs/tutorial/img/initial-view.png) Select **Index patterns** on the next view: ![OpenSearch Dashboards - Index patterns](dashboards/docs/tutorial/img/index-patterns.png) Click on **create index pattern** at the bottom. Adding the index name that used for `data-integrator` deployment (in our example: `testing`) as an index pattern enables Dashboard access to the user’s OpenSearch space. ![OpenSearch Dashboards - Create index pattern](dashboards/docs/tutorial/img/create-index-pattern.png) Click on the **Next step** button, and finalize the index pattern creation. As a verification, the user’s index metadata will display. ### Add and visualize data For test purposes, a simple method could do. Like generating data from the command-line, via the OpenSearch API: ```text for ID in `seq 1 100` do curl -sk -u opensearch-client_15:8kubD7nbWYZFHPVEzIVmTyqV42I7wHb4 \ -XPUT https://10.4.151.211:9200/testing/_doc/${ID} \ -H "Content-Type: application/json" \ -d '{"test": "This is test ${ID}"}' done ``` This is how raw data gets displayed in the Dashboard ![OpenSearch Dashboards - Data](dashboards/docs/tutorial/img/data.png) ### Data Visualization OpenSearch Dashboards offers a variety of diagrams and data displays. Choose **Dashboards** in the main left-side menu, and you will be presented to the selection: ![OpenSearch Dashboards - Visualization](dashboards/docs/tutorial/img/visualization.png) ## Clean up the environment In case you may want to remove the Juju model, you should run: ```shell juju remove-model tutorial --force --timeout 1s ``` If it’s the whole Multipass instance that you would like to delete, you should execute this command: ```shell multipass delete --purge my-vm ``` ## What’s next? Congratulations! You have successfully deployed Charmed OpenSearch Dashboards, configured TLS encryption, and explored data visualization capabilities. To continue your journey with Charmed OpenSearch Dashboards: * Explore the [How-to guides](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/index.md#dashboard-how-to-index) for guidance on practical tasks * Review the [Reference documentation](https://canonical.com/data/opensearch/docs/2//dashboards/docs/reference/index.md#dashboard-reference-index) for release notes * Check out the [Charmed OpenSearch documentation](https://canonical.com/data/opensearch/docs/) for backend integration options # index.html.md # Reference The Reference section contains pages for technical specifications, release notes, and other reference material for fast lookup. - [Release notes](https://canonical.com/data/opensearch/docs/2//reference/release-notes/index.md#reference-release-notes-index) - [System requirements](https://canonical.com/data/opensearch/docs/2//reference/system-requirements.md#reference-system-requirements) - [Default alert rules](https://canonical.com/data/opensearch/docs/2//reference/alert-rules.md#ref-alert-rules) ## Charmhub reference The following tabs on Charmhub provide automatically generated, up-to-date reference data for the OpenSearch charm: - [Actions](https://charmhub.io/opensearch/actions) — Juju actions for managing the charm (e.g. `get-password`, `create-backup`, `pre-upgrade-check`). - [Integrations](https://charmhub.io/opensearch/integrations) — Integration endpoints for connecting to other charms (e.g. `opensearch_client`, `certificates`, `s3-credentials`). - [Configurations](https://charmhub.io/opensearch/configurations) — Configuration options for the charm (e.g. `profile`, `roles`, `cluster_name`). # index.html.md # Default alert rules The following alert rules are set by default in Charmed OpenSearch.
Alert Severity Notes
OpenSearchScrapeFailed critical Triggered when the prometheus scrape fails.
OpenSearchClusterRed critical Triggered when the health status of the cluster is red, meaning that principal shards are not allocated.
OpenSearchClusterYellowTemp warning Triggered when shards are still reallocating or initializing.
OpenSearchClusterYellow warning Triggered when some replicas shards are unassigned. Might require scale the application to host all shards
OpenSearchWriteRequestsRejectionJumps warning Triggered when the write request rejection is bigger than 5%. Might indicate that the node may not keep up with the indexing speed.
OpenSearchNodeDiskLowWatermarkReached warning Triggered when disks reach 85% of the capacity.
OpenSearchNodeDiskHighWatermarkReached high Triggered when disks reach 90% of the capacity.
OpenSearchJVMHeapUseHigh alert Triggered when the JVM Heap usage in a node reaches 75%.
OpenSearchHostSystemCPUHigh alert Triggered when system CPU usage in a node reaches 90%.
OpenSearchProcessCPUHigh alert Triggered when process CPU usage in a node reaches 90%.
OpenSearchThrottling warning Triggered when a cluster is throttling. Might indicate that is necessary to review indexing request rate, index lifecycle or scale the application.
OpenSearchThrottlingTooLong critical Triggered when a cluster is constantly throttling for at least 20 minutes. Might indicate that is necessary to review indexing request rate, index lifecycle or scale the application.
# index.html.md # System requirements Charmed OpenSearch is built to operate on Ubuntu together with Juju. It is generally recommended to use the latest LTS of both, as these are the versions prioritized in the charm’s software tests. By default, OpenSearch is memory and disk-intensive, but the requirements for running Charmed OpenSearch depend heavily on your use-case. This page outlines the minimum requirements to deploy Charmed OpenSearch successfully. ## Software - Ubuntu 22.04 LTS (Jammy) or later - Juju `3.6` (latest LTS) — recommended; `3.5.3+` minimum required - Older minor versions of Juju 3 may be compatible, but are not officially supported. - LXD `6.1+` ## Hardware - 16 GB of RAM. - 4 CPU cores. - At least 20 GB of available storage - `amd64` architecture #### NOTE We highly recommend using solid-state drives (SSDs) installed on the host for node storage where possible in order to avoid performance issues in your cluster because of latency or limited throughput. #### NOTE See also: [How to perform load testing](https://canonical.com/data/opensearch/docs/2//how-to/perform-load-testing.md#how-to-perform-load-testing). ## Networking - Internet access is required for downloading artifacts from the snap and charm stores - Certain network ports must be open for internal communication: See the OpenSearch documentation for [Network requirements](https://opensearch.org/docs/2.19/install-and-configure/install-opensearch/index/#network-requirements). ## Kernel parameters OpenSearch requires specific kernel parameters. Unless noted otherwise, these settings are enforced by both the `testing` and `production` profiles. On LXD, they must be set on the host machine and propagated to every container. See [Performance profiles](https://canonical.com/data/opensearch/docs/2//explanation/performance-profiles.md#explanation-performance-profiles) for details on how profiles use these parameters. | Parameter | Required value | Purpose | |--------------------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------| | `vm.swappiness` | `0` | Disables swap to prevent OpenSearch from being swapped to disk, which causes severe performance degradation. | | `vm.max_map_count` | `262144` | Required by OpenSearch for mmap-based file access to index segments. | | `fs.file-max` | `1048576` | Ensures sufficient file descriptors for large deployments with many shards and indices. Strongly recommended, but not currently enforced by the charm. | The VM charm sets `net.ipv4.tcp_retries2` automatically. The K8s charm does not; for optional pod-level tuning, see [Configure TCP retries](https://canonical.com/data/opensearch/docs/2//how-to/deploy/standard.md#how-to-deploy-tcp-retries). #### NOTE On VM (machine) deployments, the charm applies all required `sysctl` settings automatically. For instructions on how to apply these settings on the host and propagate them to containers, see [How to deploy](https://canonical.com/data/opensearch/docs/2//how-to/deploy/standard.md#how-to-deploy-standard). # index.html.md # Revision 168 release notes 24 September 2024 Charmed OpenSearch Revision 168 has been deployed to the [`2/stable` channel](https://charmhub.io/opensearch?channel=2/stable) on Charmhub. To upgrade from a previous revision of the OpenSearch charm, see [how to perform a minor upgrade](https://canonical.com/data/opensearch/docs/2//how-to/upgrade.md#how-to-minor-upgrade). ## Highlights This section goes over the features included in this release, starting with a description of major highlights, and finishing with a comprehensive list of [all other features](#rev-168-other-features). ### Large scale deployments Deploy a single OpenSearch cluster composed of multiple Juju applications. Each application executes any of the available roles in OpenSearch. Large deployments support a diverse range of deployment constraints, roles, and regions. - [How to set up a large deployment](https://canonical.com/data/opensearch/docs/2//how-to/deploy/large.md#how-to-deploy-large) ### Security automation Manage TLS certificates and passwords in single point with Juju integrations and rotate your TLS certificates without any downtime. - [How to enable TLS encryption](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-enable-tls-encryption) - [How to rotate TLS/CA certificates](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-rotate-tls-ca-certificates) ### Monitoring Integrate with the Canonical Observability Stack (COS) and the OpenSearch Dashboards charm to monitor operational performance and visualize stored data across all clusters. - [How to enable monitoring](https://canonical.com/data/opensearch/docs/2//how-to/monitoring.md#how-to-monitoring) - [OpenSearch Dashboards: Deploy, connect, and scale](https://canonical.com/data/opensearch/docs/2//dashboards/docs/how-to/deploy-connect-scale.md#dashboard-how-to-deploy-connect-scale) ### Backups Backup and restore with an Amazon S3-compatible storage backend. - [How to configure S3 storage](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/configure-s3.md#how-to-back-up-configure-s3) - [How to create a backup](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-create-a-backup) ### Other features - **Automated rolling restart** - **Automated minor upgrade** of OpenSearch version - **Automated deployment** for single and multiple clusters - **Backup and restore** for single and multiple clusters - **User management** and automated user and index setup with the [Data Integrator charm](https://charmhub.io/data-integrator) - **TLS encryption** (HTTP and transport layers) and certificate rotation - **Observability** of OpenSearch clusters and operational tooling via COS and the [OpenSearch Dashboards charm](https://charmhub.io/opensearch-dashboards) - **Plugin management**: Index State Management, KNN and MLCommons - **OpenSearch security patching** and bug-fixing mechanisms For a detailed list of commits throughout all revisions, check our [GitHub Releases](https://github.com/canonical/opensearch-operator/releases). ## Requirements and compatibility - Juju `v3.5.3+` - Older minor versions of Juju 3 may be compatible, but are not officially supported. - LXD `v6.1` - Older LXD versions may be compatible, but are not officially supported. - Integration with a TLS charm - [`self-signed-certificates`](https://charmhub.io/self-signed-certificates) revision 155+ or [`manual-tls-certificates`](https://charmhub.io/manual-tls-certificates) revision 108+ See the [system requirements page](https://canonical.com/data/opensearch/docs/2//reference/system-requirements.md#reference-system-requirements) for more information about software and hardware prerequisites. ## Integrations Like all Juju charms, OpenSearch can easily integrate with other charms by implementing common interfaces/endpoints. OpenSearch can be seamlessly integrated out of the box with: - [TLS certificates charms](https://charmhub.io/topics/security-with-x-509-certificates) - **Note**: Charmed OpenSearch supports integration with [tls-certificates library](https://charmhub.io/tls-certificates-interface/libraries/tls_certificates) `v2` or higher. - See also: [How to enable TLS](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-enable-tls-encryption) - [COS Lite](https://charmhub.io/cos-lite) - [OpenSearch Dashboards](https://charmhub.io/opensearch-dashboards) - [Data Integrator](https://charmhub.io/data-integrator) - [S3 Integrator](https://charmhub.io/s3-integrator) See the [Integrations page](https://charmhub.io/opensearch/integrations) for a list of all interfaces and compatible charms. ## Software contents This charm is based on the Canonical [opensearch-snap](https://github.com/canonical/opensearch-snap). It packages: - OpenSearch v2.17.0 - OpenJDK `v21` ## Known issues and limitations The following issues are known and scheduled to be fixed in the next maintenance release. - We currently do not support node role repurposing from cluster manager to a non cluster manager - Storage re-attachment from previous clusters is not currently automated. For manual instructions, see the how-to guide [How to recover from attached storage](https://canonical.com/data/opensearch/docs/2//how-to/persistent-storage.md#how-to-persistent-storage). ## Join the community Charmed OpenSearch is an official distribution of OpenSearch . It’s an open-source project that welcomes community contributions, suggestions, fixes and constructive feedback. - Raise an issue or feature request in the [GitHub repository](https://github.com/canonical/opensearch-operator/issues). - Meet the community and chat with us in our [Matrix channel](https://matrix.to/#/#charmhub-data-platform:ubuntu.com) or [leave a comment](https://discourse.charmhub.io/t/charmed-opensearch-documentation/9729). - See the Charmed OpenSearch [contribution guidelines](https://github.com/canonical/opensearch-operator/blob/2/edge/CONTRIBUTING.md) on GitHub and read the Ubuntu Community’s [Code of Conduct](https://ubuntu.com/community/code-of-conduct). # index.html.md # Release notes Release notes for Charmed OpenSearch, summarising new features, bug fixes and backwards-incompatible changes in each version. See also the [guide on how to perform minor upgrades](https://canonical.com/data/opensearch/docs/2//how-to/upgrade.md#how-to-minor-upgrade). ## Releases Stable releases are published in the `2/stable` channel of CharmHub, where `2` indicates the major version of OpenSearch. - [Revision 168](https://canonical.com/data/opensearch/docs/2//reference/release-notes/revision-168.md#reference-release-notes-revision-168) - September 2024 - [Revision 315](https://canonical.com/data/opensearch/docs/2//reference/release-notes/revision-315.md#reference-release-notes-revision-315) - December 2025 ## Release policy and schedule At this time, there is no fixed release cadence. Updates will be driven by development progress, user feedback, and testing milestones. We will share release announcements and updates on the [community forum](https://discourse.charmhub.io/tags/c/announcements-and-community/33/charmed-opensearch) as they become available. # index.html.md # Revision 315 December 16, 2025 This release of the OpenSearch Operator adds support for Ubuntu 24.04 LTS (Noble Numbat), upgrades OpenSearch and OpenSearch Dashboards version to 2.19.4, and introduces significant features including OAuth and JWT authentication, and full Terraform support. [Charmhub](https://charmhub.io/opensearch) | [Deploy guide](https://canonical.com/data/opensearch/docs/2//how-to/deploy/standard.md#how-to-deploy-standard) | [Upgrade instructions](https://canonical.com/data/opensearch/docs/2//how-to/upgrade.md#how-to-minor-upgrade) | [System requirements](https://canonical.com/data/opensearch/docs/2//reference/system-requirements.md#reference-system-requirements) ## Charmed OpenSearch ### Features - [[DPE-6091](https://warthogs.atlassian.net/browse/DPE-6091)] Add OAuth integration ([PR #612](https://github.com/canonical/opensearch-operator/pull/612)) - [[DPE-8017](https://warthogs.atlassian.net/browse/DPE-8017)] Add JWT authentication ([PR #696](https://github.com/canonical/opensearch-operator/pull/696)) - [[DPE-6534](https://warthogs.atlassian.net/browse/DPE-6534)] Add Terraform charm and product modules for simple & large deployments ([PR #545](https://github.com/canonical/opensearch-operator/pull/545)) - [[DPE-4115](https://warthogs.atlassian.net/browse/DPE-4115)] Add OpenSearch Performance Profiles ([PR #466](https://github.com/canonical/opensearch-operator/pull/466)) - Upgrade OpenSearch to 2.19.4 ([PR #745](https://github.com/canonical/opensearch-operator/pull/745)) - Migrate the documentation to ReadTheDocs ([PR #708](https://github.com/canonical/opensearch-operator/pull/708)) ### Bug fixes - [[DPE-7996](https://warthogs.atlassian.net/browse/DPE-7996)] Fix OpenSearchStartTimeoutError being raised all the time ([PR #686](https://github.com/canonical/opensearch-operator/pull/686)) - [[DPE-8408](https://warthogs.atlassian.net/browse/DPE-8408)] Isolate OpenSearchNotFullyReadyError and add retry mechanism ([PR #704](https://github.com/canonical/opensearch-operator/pull/704)) - [[DPE-7828](https://warthogs.atlassian.net/browse/DPE-7828)] Fix main orchestrator demotion ([PR #689](https://github.com/canonical/opensearch-operator/pull/689)) - [[DPE-6877](https://warthogs.atlassian.net/browse/DPE-6877)] Fix failover bootstrapping cluster ([PR #629](https://github.com/canonical/opensearch-operator/pull/629)) - [[DPE-6869](https://warthogs.atlassian.net/browse/DPE-6869)] Fix starting up after scale down to 0 ([PR #635](https://github.com/canonical/opensearch-operator/pull/635)) - [[DPE-7559](https://warthogs.atlassian.net/browse/DPE-7559)] Fix leader-elected hook for non-leader units ([PR #661](https://github.com/canonical/opensearch-operator/pull/661)) - [[DPE-6422](https://warthogs.atlassian.net/browse/DPE-6422)] Fix upgrade path with snap rev. 65 ([PR #553](https://github.com/canonical/opensearch-operator/pull/553)) - [[DPE-7076](https://warthogs.atlassian.net/browse/DPE-7076)] Data node fails to start ([PR #671](https://github.com/canonical/opensearch-operator/pull/671)) - [[DPE-7545](https://warthogs.atlassian.net/browse/DPE-7545)] Fix s3-uri-style handling ([PR #654](https://github.com/canonical/opensearch-operator/pull/654)) - [[DPE-5460](https://warthogs.atlassian.net/browse/DPE-5460)] Restart data only nodes without lock ([PR #460](https://github.com/canonical/opensearch-operator/pull/460)) - [[DPE-5671](https://warthogs.atlassian.net/browse/DPE-5671)] Logic change in how check if CA rotation in complete in cluster ([PR #486](https://github.com/canonical/opensearch-operator/pull/486)) - Only data node ignores lock if restarting ([PR #762](https://github.com/canonical/opensearch-operator/pull/762)) - Fixing Terraform Flaky Tests due to Limited storage on github runners ([PR #765](https://github.com/canonical/opensearch-operator/pull/765)) - Fix CA rotation edge case on small deployments ([PR #767](https://github.com/canonical/opensearch-operator/pull/767)) - Set blocked message on verify repository fail ([PR #769](https://github.com/canonical/opensearch-operator/pull/769)) - Clear missing relations ([PR #763](https://github.com/canonical/opensearch-operator/pull/763)) - Remove verify repository from requirer error checks ([PR #772](https://github.com/canonical/opensearch-operator/pull/772)) ### Other improvements - [[DPE-5830](https://warthogs.atlassian.net/browse/DPE-5830)] Bump to Ubuntu 24.04 LTS (Noble Numbat) ([PR #495](https://github.com/canonical/opensearch-operator/pull/495)) - [[DPE-6598](https://warthogs.atlassian.net/browse/DPE-6598)] Move to OpenSearch v2.18 ([PR #563](https://github.com/canonical/opensearch-operator/pull/563)) - [[DPE-4196](https://warthogs.atlassian.net/browse/DPE-4196)] Plugin Management Refactor ([PR #435](https://github.com/canonical/opensearch-operator/pull/435)) - [[DPE-6922](https://warthogs.atlassian.net/browse/DPE-6922)] Integrate TiCS ([PR #605](https://github.com/canonical/opensearch-operator/pull/605)) - [[DPE-6285](https://warthogs.atlassian.net/browse/DPE-6285)] Update poetry to v2 ([PR #527](https://github.com/canonical/opensearch-operator/pull/527)) - [[DPE-5444](https://warthogs.atlassian.net/browse/DPE-5444)] Bump to ops 2.19 ([PR #575](https://github.com/canonical/opensearch-operator/pull/575)) - [[DPE-7560](https://warthogs.atlassian.net/browse/DPE-7560)] Remove wrong default base path for snapshots repositories ([PR #658](https://github.com/canonical/opensearch-operator/pull/658)) - [[DPE-6878](https://warthogs.atlassian.net/browse/DPE-6878)] Stabilize upgrade tests ([PR #626](https://github.com/canonical/opensearch-operator/pull/626)) - [[DPE-6787](https://warthogs.atlassian.net/browse/DPE-6787)] Add smoke test ([PR #588](https://github.com/canonical/opensearch-operator/pull/588)) - [[DPE-5667](https://warthogs.atlassian.net/browse/DPE-5667)] Move away from ops_test’s wait_for_idle to wait_until ([PR #479](https://github.com/canonical/opensearch-operator/pull/479)) - Replace Discourse docs with a charm description on Charmhub ([PR #760](https://github.com/canonical/opensearch-operator/pull/760)) - Refactor code and verify repository after all units save credentials ([PR #774](https://github.com/canonical/opensearch-operator/pull/774)) - Rework Plugins ([PR #775](https://github.com/canonical/opensearch-operator/pull/775)) - Rework Snapshots ([PR #718](https://github.com/canonical/opensearch-operator/pull/718)) ## Charmed OpenSearch Dashboards ### Features - [[DPE-5867](https://warthogs.atlassian.net/browse/DPE-5867)] Add terraform module ([PR #134](https://github.com/canonical/opensearch-dashboards-operator/pull/134)) - [[DPE-6091](https://warthogs.atlassian.net/browse/DPE-6091)] Add oauth integration ([PR #180](https://github.com/canonical/opensearch-dashboards-operator/pull/180)) - [[DPE-6936](https://warthogs.atlassian.net/browse/DPE-6936)] Update to 2.19.1 ([PR #182](https://github.com/canonical/opensearch-dashboards-operator/pull/182)) - [[DPE-8018](https://warthogs.atlassian.net/browse/DPE-8018)] Add JWT authentication ([PR #217](https://github.com/canonical/opensearch-dashboards-operator/pull/217)) - [[DPE-5832](https://warthogs.atlassian.net/browse/DPE-5832)][[DPE-6252](https://warthogs.atlassian.net/browse/DPE-6252)] Add 24.04 LTS support ([PR #138](https://github.com/canonical/opensearch-dashboards-operator/pull/138)) - Add new Prometheus Rules using the python exporter ([PR #159](https://github.com/canonical/opensearch-dashboards-operator/pull/159)) - Bump version to 2.19.2 ([PR #198](https://github.com/canonical/opensearch-dashboards-operator/pull/198)) - Release 2.19.4 ([PR #224](https://github.com/canonical/opensearch-dashboards-operator/pull/224)) ### Bug fixes - [[DPE-7774](https://warthogs.atlassian.net/browse/DPE-7774)] Raise if error on install ([PR #214](https://github.com/canonical/opensearch-dashboards-operator/pull/214)) - [[DPE-7002](https://warthogs.atlassian.net/browse/DPE-7002)] Update opensearch health checks ([PR #215](https://github.com/canonical/opensearch-dashboards-operator/pull/215)) - [[DPE-7928](https://warthogs.atlassian.net/browse/DPE-7928)] Block if we fail to get the provider info from the oauth relation ([PR #218](https://github.com/canonical/opensearch-dashboards-operator/pull/218)) - [[DPE-6600](https://warthogs.atlassian.net/browse/DPE-6600)] Move away from `libjuju` and `add_machine` ([PR #167](https://github.com/canonical/opensearch-dashboards-operator/pull/167)) - [[DPE-6598](https://warthogs.atlassian.net/browse/DPE-6598)][[DPE-6786](https://warthogs.atlassian.net/browse/DPE-6786)] Update to 2.18 and fix CI ([PR #169](https://github.com/canonical/opensearch-dashboards-operator/pull/169)) - [[DPE-5699](https://warthogs.atlassian.net/browse/DPE-5699)] Update CI workflow versions, add Juju 3.6, and remove build wrapper ([PR #124](https://github.com/canonical/opensearch-dashboards-operator/pull/124)) - Fix branch release on CI ([PR #123](https://github.com/canonical/opensearch-dashboards-operator/pull/123)) - Remove dependency breaking ccc-hub ([PR #136](https://github.com/canonical/opensearch-dashboards-operator/pull/136)) - Rollback Cache use ([PR #139](https://github.com/canonical/opensearch-dashboards-operator/pull/139)) ### Other improvements - [[DPE-6922](https://warthogs.atlassian.net/browse/DPE-6922)] Integrate TiCS ([PR #176](https://github.com/canonical/opensearch-dashboards-operator/pull/176)) - [[DPE-4307](https://warthogs.atlassian.net/browse/DPE-4307)] HA process interrupt tests ([PR #114](https://github.com/canonical/opensearch-dashboards-operator/pull/114)) - [[DPE-6876](https://warthogs.atlassian.net/browse/DPE-6876)] Migration to spread ([PR #178](https://github.com/canonical/opensearch-dashboards-operator/pull/178)) - [[DPE-6217](https://warthogs.atlassian.net/browse/DPE-6217)] Remove channel + revision mentions ([PR #145](https://github.com/canonical/opensearch-dashboards-operator/pull/145)) - [[DPE-6536](https://warthogs.atlassian.net/browse/DPE-6536)] Improve terraform charm module ([PR #164](https://github.com/canonical/opensearch-dashboards-operator/pull/164)) - [[DPE-6687](https://warthogs.atlassian.net/browse/DPE-6687)][[DPE-6690](https://warthogs.atlassian.net/browse/DPE-6690)][[DPE-6691](https://warthogs.atlassian.net/browse/DPE-6691)] Extend TF charm modules ([PR #168](https://github.com/canonical/opensearch-dashboards-operator/pull/168)) - [[DPE-6858](https://warthogs.atlassian.net/browse/DPE-6858)] Add expose support in Terraform([PR #173](https://github.com/canonical/opensearch-dashboards-operator/pull/173)) - [[DPE-5704](https://warthogs.atlassian.net/browse/DPE-5704)] Update COS to use jammy series ([PR #125](https://github.com/canonical/opensearch-dashboards-operator/pull/125)) - Add alert if an OpenSearch Dashboards scrape fails ([PR #140](https://github.com/canonical/opensearch-dashboards-operator/pull/140)) - Add `promtool` check and test in the CI ([PR #142](https://github.com/canonical/opensearch-dashboards-operator/pull/142)) - Both `ci.yaml` and `test_prometheus_rules.yaml` were using same lock ([PR #146](https://github.com/canonical/opensearch-dashboards-operator/pull/146)) - Migrate to charmcraft 3 poetry plugin ([PR #148](https://github.com/canonical/opensearch-dashboards-operator/pull/148)) - Finish charmcraft 3 migration ([PR #150](https://github.com/canonical/opensearch-dashboards-operator/pull/150)) - Switch charmcraft to latest/candidate ([PR #151](https://github.com/canonical/opensearch-dashboards-operator/pull/151)) - Use single (cached) build for tests & release ([PR #155](https://github.com/canonical/opensearch-dashboards-operator/pull/155)) - Remove unused dependencies from main and charm-libs groups ([PR #156](https://github.com/canonical/opensearch-dashboards-operator/pull/156)) - Use stage instead of prime in charmcraft files part ([PR #158](https://github.com/canonical/opensearch-dashboards-operator/pull/158)) - Switch to charmcraft stable ([PR #162](https://github.com/canonical/opensearch-dashboards-operator/pull/162)) - Add revision to the grafana dashboard title ([PR #163](https://github.com/canonical/opensearch-dashboards-operator/pull/163)) - Update tls channel ([PR #170](https://github.com/canonical/opensearch-dashboards-operator/pull/170)) - Update tls lib ([PR #127](https://github.com/canonical/opensearch-dashboards-operator/pull/127)) - Add config profile=testing ([PR #128](https://github.com/canonical/opensearch-dashboards-operator/pull/128)) - Remove get-password ([PR #130](https://github.com/canonical/opensearch-dashboards-operator/pull/130)) ## Compatibility | Charm | Revision | Hardware architecture | OpenSearch version | Minimum Juju version | Artifacts | |-------------------------------|------------------------------------------------------------------------------|-------------------------|---------------------------------------------------------------------------------|------------------------|-----------------------------------------------------------------| | Charmed OpenSearch | [315](https://github.com/canonical/opensearch-operator/tree/rev315) | AMD64 | [v2.19.4](https://github.com/opensearch-project/OpenSearch/releases/tag/2.19.4) | 3.5+ | Snap: [revision 98](https://snapcraft.io/opensearch) | | Charmed OpenSearch Dashboards | [60](https://github.com/canonical/opensearch-dashboards-operator/tree/rev60) | AMD64 | [v2.19.4](https://github.com/opensearch-project/OpenSearch/releases/tag/2.19.4) | 3.5+ | Snap: [revision 54](https://snapcraft.io/opensearch-dashboards) | # index.html.md # Tutorial This tutorial is designed to help you learn how to deploy Charmed OpenSearch and become familiar with its available operations. ## What you’ll need While this tutorial intends to guide you as you deploy Charmed OpenSearch for the first time, it will be most beneficial if: - You have some experience using a Linux-based CLI - You are familiar with OpenSearch concepts such as indices and users. - To learn more, see the official [OpenSearch documentation](https://opensearch.org/docs/2.19/about/) - Your computer fulfills the [minimum system requirements](https://canonical.com/data/opensearch/docs/2//reference/system-requirements.md#reference-system-requirements) ## What you’ll do | Step | Details | |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | 1. [**Set up the environment**](https://canonical.com/data/opensearch/docs/2//tutorial/1-set-up-the-environment.md#tutorial-1-set-up-the-environment) | Set up a cloud environment for your deployment with LXD | | 2. [**Deploy OpenSearch**](https://canonical.com/data/opensearch/docs/2//tutorial/2-deploy-opensearch.md#tutorial-2-deploy-opensearch) | Learn how to deploy OpenSearch with Juju | | 3. [**Enable encryption**](https://canonical.com/data/opensearch/docs/2//tutorial/3-enable-encryption.md#tutorial-3-enable-encryption) | Enable security in your deployment by integrating with a TLS certificates operator | | 4. [**Integrate with a client application**](https://canonical.com/data/opensearch/docs/2//tutorial/4-integrate-with-a-client-application.md#tutorial-4-integrate-with-a-client-application) | Learn how to integrate a client app with OpenSearch and manage users | | 5. [**Manage passwords**](https://canonical.com/data/opensearch/docs/2//tutorial/5-manage-passwords.md#tutorial-5-manage-passwords) | Learn about password management and rotation | | 6. [**Scale horizontally**](https://canonical.com/data/opensearch/docs/2//tutorial/6-scale-horizontally.md#tutorial-6-scale-horizontally) | Scale your application by adding or removing Juju units | | 7. [**Clean up the environment**](https://canonical.com/data/opensearch/docs/2//tutorial/7-clean-up-the-environment.md#tutorial-7-clean-up-the-environment) | Remove your OpenSearch deployment and Juju to free your machine’s resources | # index.html.md # 5. Manage passwords > [Charmed OpenSearch Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/index.md#tutorial-index) > 5. Manage passwords When we accessed OpenSearch earlier in this tutorial, we needed to include a password in the HTTP request. Over time it is a good practice to change the password frequently. Here we will go through setting and changing the password for the admin user. ## Retrieve the admin password As previously mentioned, the admin credentials (password + the ca chain used to generate the admin client certificate) can be retrieved by running the `get-password` action on the Charmed OpenSearch application: ```bash juju run opensearch/leader get-password ``` Running the command above should output something like: ```yaml Running operation 7 with 1 task - task 8 on unit-opensearch-0 Waiting for task 8... ca-chain: |- -----BEGIN CERTIFICATE----- MIIDPzCCAiegAwIB... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDOzCCAiOgAwIB... -----END CERTIFICATE----- password: IbmPfpZthaWMQhxhtm9XyqmSDYGivBpC username: admin ``` The admin password is under the result: `password`. ## Rotate the admin password You can change the admin password to a new random and generated password by running: ```shell juju run opensearch/leader set-password ``` #### NOTE This action can only be run from the leader unit. Running the command should output: ```yaml Running operation 9 with 1 task - task 10 on unit-opensearch-0 Waiting for task 10... admin-password: aW1kMu2pO4GGdw52nfrYHAayu8rn4nn9 ``` The admin password is under the result: `admin-password`. It should be different from your previous password. You can test this password works correctly using the same requests you used during the [integrate with a client section](https://canonical.com/data/opensearch/docs/2//tutorial/4-integrate-with-a-client-application.md#tutorial-4-integrate-with-a-client-application) of this tutorial. ## Set the admin password You can change the admin password to a specific password by entering: ```bash juju run opensearch/leader set-password password= ``` Running the command should output: ```yaml Running operation 11 with 1 task - task 12 on unit-opensearch-0 Waiting for task 12... admin-password: ``` The admin password under the result: `admin-password` should match whatever you passed in when you entered the command. ## Set TLS Private Key TLS private keys are used for certificate signing requests and should be recycled in the same way as passwords. There are three types of private keys available to be updated on this charm, and they are as follows: - `"app-admin"` is the key used for requesting a certificate with a CSR for the admin user and cluster administration-related operations. - Must only be set on the leader unit. - `"unit-transport"` is the key used for requesting, for the target unit, a certificate with a CSR for the transport layer (node-to-node communication). - `"unit-http"` is the key used for requesting, for the target unit, a certificate with a CSR for the HTTP layer. This is used for client-to-node communication. To change a private key to a random value, run the following command, setting `category` equal to your preferred type of private key: ```bash juju run opensearch/leader set-tls-private-key category= ``` Running the command should output: ```yaml Running operation 13 with 1 task - task 14 on unit-opensearch-0 Waiting for task 14... ``` No certificate data is presented in the results of this action. To set the key to a specific value run the following command: ```bash juju run opensearch/leader set-tls-private-key category= key= ``` Running the command should output: ```yaml Running operation 15 with 1 task - task 16 on unit-opensearch-0 Waiting for task 16... ``` If the key you intend to set has a passphrase, set it like so ```bash juju run opensearch/leader set-tls-private-key category= password= key= ``` Running the command should output: ```yaml Running operation 17 with 1 task - task 18 on unit-opensearch-0 Waiting for task 18... ``` #### NOTE **Next step**: [6. Scale horizontally](https://canonical.com/data/opensearch/docs/2//tutorial/6-scale-horizontally.md#tutorial-6-scale-horizontally) # index.html.md # 1. Set up the environment > [Charmed OpenSearch Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/index.md#tutorial-index) > 1. Set up the environment In this step, we will set up a development environment with the required components for deploying Charmed OpenSearch. #### NOTE Before you start, make sure your machine meets the [minimum system requirements](https://canonical.com/data/opensearch/docs/2//reference/system-requirements.md#reference-system-requirements). ## Set up LXD The simplest way to get started with Charmed OpenSearch is to set up a local LXD cloud. [LXD](https://documentation.ubuntu.com/lxd/en/latest/) is a system container and virtual machine manager that comes pre-installed on Ubuntu. Juju interfaces with LXD to control the containers on which Charmed OpenSearch runs. Verify if your Ubuntu system already has LXD installed with the command `which lxd`. If there is no output, then install LXD with ```shell sudo snap install lxd ``` After installation, `lxd init` is run to perform post-installation tasks. For this tutorial, the default parameters are preferred and the network bridge should be set to have no IPv6 addresses since Juju does not support IPv6 addresses with LXD: ```shell lxd init --auto lxc network set lxdbr0 ipv6.address none ``` You can list all LXD containers by executing the command `lxc list`. At this point in the tutorial, none should exist, so you’ll only see this as output: ```text +------+-------+------+------+------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +------+-------+------+------+------+-----------+ ``` ## Set up Juju [Juju](https://juju.is/docs/juju) is an Operator Lifecycle Manager (OLM) for clouds, bare metal, LXD or Kubernetes. We will be using it to deploy and manage Charmed OpenSearch. As with LXD, Juju is installed using a snap package: ```shell sudo snap install juju --channel 3/stable ``` Juju already has a built-in knowledge of LXD and how it works, so there is no additional setup or configuration needed, however, because Juju 3.x is a [strictly confined snap](https://snapcraft.io/docs/explanation/security/snap-confinement), and is not allowed to create a `~/.local/share` directory, we need to create it manually. ```shell mkdir -p ~/.local/share ``` To list the clouds available to Juju, run the following command: ```shell juju clouds ``` The output will look as follows: ```text Clouds available on the client: Cloud Regions Default Type Credentials Source Description localhost 1 localhost lxd 0 built-in LXD Container Hypervisor ``` A controller will be used to deploy and control Charmed OpenSearch. Run the following command to bootstrap a Juju controller named `opensearch-demo` on LXD: ```shell juju bootstrap localhost opensearch-demo ``` This bootstrapping process can take several minutes depending on your system resources. The Juju controller exists within an LXD container. You can verify this by entering the command `lxc list`. This will output the following: ```text +---------------+---------+-----------------------+------+-----------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +---------------+---------+-----------------------+------+-----------+-----------+ | juju- | RUNNING | 10.105.164.235 (eth0) | | CONTAINER | 0 | +---------------+---------+-----------------------+------+-----------+-----------+ ``` where `` is a unique combination of numbers and letters such as `9d7e4e-0` Set up a unique model for this tutorial named `tutorial`: ```shell juju add-model tutorial ``` You can now view the model you created above by entering the command `juju status` into the command line. You should see the following: ```text Model Controller Cloud/Region Version SLA Timestamp tutorial opensearch-demo localhost/localhost 3.6.23 unsupported 11:26:13Z ``` ## Set kernel parameters Before deploying Charmed OpenSearch, we need to set some [kernel parameters](https://www.kernel.org/doc/Documentation/sysctl/vm.txt). These are requirements for OpenSearch to function correctly. Since we are using LXD containers to deploy our charm, and containers share a kernel with their host, we need to set these kernel parameters on the host machine. We will save the default values, change them to the optimal values for OpenSearch, and add the parameters to the Juju model’s configuration. ### Get default values First, we need to make note of the current parameters of the kernel because we will need to reset them after the tutorial (although rebooting your machine will also do the trick). Let’s run `sysctl` and filter the output for the two specific parameters that we will be changing: ```shell sudo sysctl -a | grep -E 'swappiness|max_map_count' ``` This command should return something like the following: ```text vm.max_map_count = 262144 vm.swappiness = 60 ``` Make note of the above variables so that you can reset them later to their original values. Using the host machine outside of this tutorial without resetting these kernel parameters manually or rebooting may have an impact on the host machine’s performance. ### Set parameters on the host machine Set the kernel parameters to the recommended values for OpenSearch with the following commands: ```shell sudo tee -a /etc/sysctl.conf > /dev/null < cloudinit-userdata.yaml cloudinit-userdata: | postruncmd: - echo 'vm.max_map_count=262144' >> /etc/sysctl.conf - echo 'vm.swappiness=0' >> /etc/sysctl.conf - echo 'fs.file-max=1048576' >> /etc/sysctl.conf - sysctl -p EOF juju model-config --file=./cloudinit-userdata.yaml ``` #### NOTE Keep each `postruncmd` entry as a **string**. Cloud-init runs string entries through a shell, so the `>>` redirection works. Entries written as a YAML list are passed straight to `execve(3)` with no shell, so `>>` would become a literal argument to `echo` instead of appending to the file. #### NOTE **Next step:** [2. Deploy OpenSearch](https://canonical.com/data/opensearch/docs/2//tutorial/2-deploy-opensearch.md#tutorial-2-deploy-opensearch). # index.html.md # 7. Clean up the environment > [Charmed OpenSearch Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/index.md#tutorial-index) > 7. Clean up the environment In this tutorial, we’ve successfully: - Deployed OpenSearch on LXD - Enabled TLS - Integrated with a client application - Rotated user credentials - Scaled our deployment You may now keep your OpenSearch deployment running to continue experimenting or remove it entirely to free up resources on your machine. ## Remove Charmed OpenSearch #### WARNING When you remove Charmed OpenSearch as shown below, you will lose all the data in your cluster. To remove Charmed OpenSearch and the model it is hosted on, run this command: ```bash juju destroy-model tutorial --destroy-storage --force --no-wait ``` The next step is to remove the Juju controller. You can see all of the available controllers by entering `juju controllers`. To remove the controller created for this tutorial, enter: ```bash juju destroy-controller opensearch-demo ``` Then, don’t forget to delete the Juju model configuration file. ```bash rm cloudinit-userdata.yaml ``` ## Remove Juju #### WARNING When you remove Juju as shown below you will lose access to any other applications you have hosted on Juju. To remove Juju altogether, enter: ```bash sudo snap remove juju --purge ``` ## Remove LXD #### WARNING When you remove LXD as shown below you will lose access to any other applications you have hosted on LXD. You can list all your currently running LXD container with `lxc list`. To uninstall Juju, enter: ```bash sudo snap remove lxd --purge ``` ## Reset the Kernel parameters #### WARNING In the following command, use the values you saved during step 1 -> Get default values. If you did not save those values, use the second reset option. Leaving the custom kernel parameters outside of this tutorial scope can impact the host machine’s performance. To reset them, you can either: - Reboot your computer - Set your original parameters with the following command: ```shell sudo tee -a /etc/sysctl.conf > /dev/null < ## What next? - Check out other charms on [charmhub.io](https://charmhub.io/) - Read about [High Availability Best Practices](https://canonical.com/blog/database-high-availability) - [Report](https://github.com/canonical/opensearch-operator/issues) any problems you encountered - [Give us your feedback](https://matrix.to/#/#charmhub-data-platform:ubuntu.com) - [Contribute to the code base](https://github.com/canonical/opensearch-operator) # index.html.md # 3. Enable encryption > [Charmed OpenSearch Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/index.md#tutorial-index) > 3. Enable TLS encryption [Transport Layer Security (TLS)](https://en.wikipedia.org/wiki/Transport_Layer_Security) is a protocol used to encrypt data exchanged between two applications. Essentially, it secures data transmitted over a network. Typically, enabling TLS internally within a highly available database or between a highly available database and client/server applications requires a high level of expertise. This has all been encoded into Charmed OpenSearch so that configuring TLS requires minimal effort on your end. TLS is enabled by integrating Charmed OpenSearch with the [Self Signed Certificates Charm](https://charmhub.io/self-signed-certificates). This charm centralizes TLS certificate management consistently and handles operations like providing, requesting, and renewing TLS certificates. In this section, you will learn how to enable security in your OpenSearch deployment using TLS encryption. #### NOTE **[Self-signed certificates](https://en.wikipedia.org/wiki/Self-signed_certificate) are not recommended for a production environment.**
\\\\ Check the [X.509 certificates topic](https://charmhub.io/topics/security-with-x-509-certificates) for an overview of the TLS certificates charms available. ## Configure TLS Before enabling TLS on Charmed OpenSearch we must first deploy the `self-signed-certificates` charm: ```shell juju deploy self-signed-certificates --channel 1/stable --config ca-common-name="Tutorial CA" ``` Wait until `self-signed-certificates` is active. Use `juju status --watch 1s` to monitor the progress. ```text Model Controller Cloud/Region Version SLA Timestamp tutorial opensearch-demo localhost/localhost 3.6.23 unsupported 12:39:22Z App Version Status Scale Charm Channel Rev Exposed Message opensearch blocked 3 opensearch 2/stable 344 no Missing TLS relation with this cluster. self-signed-certificates active 1 self-signed-certificates 1/stable 586 no Unit Workload Agent Machine Public address Ports Message opensearch/0 blocked idle 0 10.95.38.94 Missing TLS relation with this cluster. opensearch/1 blocked idle 1 10.95.38.139 Missing TLS relation with this cluster. opensearch/2* blocked idle 2 10.95.38.212 Missing TLS relation with this cluster. self-signed-certificates/0* active idle 3 10.95.38.54 Machine State Address Inst id Base AZ Message 0 started 10.95.38.94 juju-be3883-0 ubuntu@24.04 Running 1 started 10.95.38.139 juju-be3883-1 ubuntu@24.04 Running 2 started 10.95.38.212 juju-be3883-2 ubuntu@24.04 Running 3 started 10.95.38.54 juju-be3883-3 ubuntu@24.04 Running ``` ## Integrate with OpenSearch To enable TLS on Charmed OpenSearch, you must integrate (also known as “relate”) the two applications. We will go over integrations in more detail in the [next page](https://canonical.com/data/opensearch/docs/2//tutorial/4-integrate-with-a-client-application.md#tutorial-4-integrate-with-a-client-application) of this tutorial. To integrate `self-signed-certificates` with `opensearch`, run the following command: ```shell juju integrate opensearch:certificates self-signed-certificates ``` The OpenSearch service will start. This might take some time. Once done, you can see the new integrations with `juju status --relations`. ```text Model Controller Cloud/Region Version SLA Timestamp tutorial opensearch-demo localhost/localhost 3.6.23 unsupported 12:41:22Z App Version Status Scale Charm Channel Rev Exposed Message opensearch active 3 opensearch 2/stable 344 no self-signed-certificates active 1 self-signed-certificates 1/stable 586 no Unit Workload Agent Machine Public address Ports Message opensearch/0 active idle 0 10.95.38.94 9200/tcp opensearch/1 active idle 1 10.95.38.139 9200/tcp opensearch/2* active idle 2 10.95.38.212 9200/tcp self-signed-certificates/0* active idle 3 10.95.38.54 Machine State Address Inst id Base AZ Message 0 started 10.95.38.94 juju-be3883-0 ubuntu@24.04 Running 1 started 10.95.38.139 juju-be3883-1 ubuntu@24.04 Running 2 started 10.95.38.212 juju-be3883-2 ubuntu@24.04 Running 3 started 10.95.38.54 juju-be3883-3 ubuntu@24.04 Running Integration provider Requirer Interface Type Message opensearch:node-lock-fallback opensearch:node-lock-fallback node_lock_fallback peer opensearch:opensearch-peers opensearch:opensearch-peers opensearch_peers peer opensearch:upgrade-version-a opensearch:upgrade-version-a upgrade peer self-signed-certificates:certificates opensearch:certificates tls-certificates regular ``` Notice the last relation: `self-signed-certificates:certificates opensearch:certificates tls-certificates regular`. #### NOTE **Next step:** [4. Integrate with a client application](https://canonical.com/data/opensearch/docs/2//tutorial/4-integrate-with-a-client-application.md#tutorial-4-integrate-with-a-client-application) # index.html.md # 2. Deploy OpenSearch > [Charmed OpenSearch Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/index.md#tutorial-index) > 2. Deploy OpenSearch Charmed OpenSearch supports performance profiles. It is recommended in a single host deployment with LXD to use the `testing` profile, which will only consume 1G RAM per container. To deploy Charmed OpenSearch, run the following command: ```shell juju deploy opensearch -n 3 ``` #### NOTE The `-n` flag is optional and specifies the number of units to deploy. In this case, we are deploying three units of Charmed OpenSearch. We recommend deploying at least three units for high availability. This command will fetch the charm from [Charmhub](https://charmhub.io/opensearch) and deploy 3 units to the LXD cloud. This process can take several minutes depending on your machine. You can track the progress by running: ```bash juju status --watch 1s ``` #### NOTE This command is useful for checking the status of your Juju model, including the applications and machines it hosts. Helpful information it displays includes IP addresses, ports, state, etc. The output of this command updates once every other second. When the application is ready, `juju status` will show something similar to the sample output below: ```text Model Controller Cloud/Region Version SLA Timestamp tutorial opensearch-demo localhost/localhost 3.6.23 unsupported 12:36:56Z App Version Status Scale Charm Channel Rev Exposed Message opensearch blocked 3 opensearch 2/stable 344 no Missing TLS relation with this cluster. Unit Workload Agent Machine Public address Ports Message opensearch/0 blocked idle 0 10.95.38.94 Missing TLS relation with this cluster. opensearch/1 blocked idle 1 10.95.38.139 Missing TLS relation with this cluster. opensearch/2* blocked idle 2 10.95.38.212 Missing TLS relation with this cluster. Machine State Address Inst id Base AZ Message 0 started 10.95.38.94 juju-be3883-0 ubuntu@24.04 Running 1 started 10.95.38.139 juju-be3883-1 ubuntu@24.04 Running 2 started 10.95.38.212 juju-be3883-2 ubuntu@24.04 Running ``` To exit the `juju status` screen, enter `Ctrl + C`. The status message `Missing TLS relation with this cluster.` is displayed because Charmed OpenSearch requires TLS to be configured before use, to ensure data is encrypted in transit for the HTTP and Transport layers. We will do this in the next step. If you see the following status message: ```text Missing requirements: vm.swappiness should be at most 0 ``` you need to [set the correct kernel parameters](https://canonical.com/data/opensearch/docs/2//tutorial/1-set-up-the-environment.md#tutorial-1-set-up-the-environment) to continue. #### NOTE **Next step:** [3. Enable TLS](https://canonical.com/data/opensearch/docs/2//tutorial/3-enable-encryption.md#tutorial-3-enable-encryption) # index.html.md # 4. Integrate with a client application > [Charmed OpenSearch Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/index.md#tutorial-index) > 4. Integrate with a client application [Integrations](https://documentation.ubuntu.com/juju/3.6/reference/relation/) (also known as “relations”) are the easiest way to connect to Charmed OpenSearch. Integrations automatically create a username, password, and database for the desired user/application, and define access permissions. ## Deploy the Data Integrator charm The best way to create a user and password for manual use (i.e. connecting to OpenSearch directly using `curl`, which is what we’ll be doing later) is to add a relation between Charmed OpenSearch and the [Data Integrator Charm](https://charmhub.io/data-integrator). Data Integrator is a bare-bones charm that allows for central management of database users, providing support for different kinds of data platform products (e.g. MongoDB, MySQL, PostgreSQL, Kafka, etc) with a consistent and robust user experience. Deploy Data Integrator as follows: ```shell juju deploy data-integrator --channel=stable --config index-name=test-index --config extra-user-roles=admin ``` The expected output: ```text Deployed "data-integrator" from charm-hub charm "data-integrator", revision 362 in channel latest/stable on ubuntu@24.04/stable ``` Wait for `juju status --watch 1s` to show: ```text Model Controller Cloud/Region Version SLA Timestamp tutorial opensearch-demo localhost/localhost 3.6.23 unsupported 12:43:22Z App Version Status Scale Charm Channel Rev Exposed Message data-integrator blocked 1 data-integrator latest/stable 362 no Please relate the data-integrator with the desired product opensearch active 3 opensearch 2/stable 344 no self-signed-certificates active 1 self-signed-certificates 1/stable 586 no Unit Workload Agent Machine Public address Ports Message data-integrator/0* blocked idle 4 10.95.38.22 Please relate the data-integrator with the desired product opensearch/0 active idle 0 10.95.38.94 9200/tcp opensearch/1 active idle 1 10.95.38.139 9200/tcp opensearch/2* active idle 2 10.95.38.212 9200/tcp self-signed-certificates/0* active idle 3 10.95.38.54 Machine State Address Inst id Base AZ Message 0 started 10.95.38.94 juju-be3883-0 ubuntu@24.04 Running 1 started 10.95.38.139 juju-be3883-1 ubuntu@24.04 Running 2 started 10.95.38.212 juju-be3883-2 ubuntu@24.04 Running 3 started 10.95.38.54 juju-be3883-3 ubuntu@24.04 Running 4 started 10.95.38.22 juju-be3883-4 ubuntu@24.04 Running Integration provider Requirer Interface Type Message data-integrator:data-integrator-peers data-integrator:data-integrator-peers data-integrator-peers peer opensearch:node-lock-fallback opensearch:node-lock-fallback node_lock_fallback peer opensearch:opensearch-peers opensearch:opensearch-peers opensearch_peers peer opensearch:upgrade-version-a opensearch:upgrade-version-a upgrade peer self-signed-certificates:certificates opensearch:certificates tls-certificates regular ``` Notice that the status of the `data-integrator` application is `blocked`. This is because it is waiting for a relation to be established with another application namely `opensearch`. ## Integrate with OpenSearch Now that the Database Integrator charm has been set up, we can relate it to Charmed OpenSearch. This will automatically create a username, password, and CA certificate for the Database Integrator charm. Integrate the two applications with: ```shell juju integrate data-integrator opensearch ``` Wait for `juju status --relations --watch 1s` to show that the `data-integrator` application is now active: Model Controller Cloud/Region Version SLA Timestamp tutorial opensearch-demo localhost/localhost 3.6.23 unsupported 12:44:43Z App Version Status Scale Charm Channel Rev Exposed Message data-integrator active 1 data-integrator latest/stable 362 no
\\\\ opensearch active 3 opensearch 2/stable 344 no
\\\\ self-signed-certificates active 1 self-signed-certificates 1/stable 586 no Unit Workload Agent Machine Public address Ports Message data-integrator/0\* active idle 4 10.95.38.22
\\\\ opensearch/0 active idle 0 10.95.38.94 9200/tcp
\\\\ opensearch/1 active idle 1 10.95.38.139 9200/tcp
\\\\ opensearch/2\* active idle 2 10.95.38.212 9200/tcp
\\\\ self-signed-certificates/0\* active idle 3 10.95.38.54 Machine State Address Inst id Base AZ Message 0 started 10.95.38.94 juju-be3883-0 ubuntu@24.04 Running 1 started 10.95.38.139 juju-be3883-1 ubuntu@24.04 Running 2 started 10.95.38.212 juju-be3883-2 ubuntu@24.04 Running 3 started 10.95.38.54 juju-be3883-3 ubuntu@24.04 Running 4 started 10.95.38.22 juju-be3883-4 ubuntu@24.04 Running Integration provider Requirer Interface Type Message data-integrator:data-integrator-peers data-integrator:data-integrator-peers data-integrator-peers peer opensearch:node-lock-fallback opensearch:node-lock-fallback node_lock_fallback peer opensearch:opensearch-client data-integrator:opensearch opensearch_client regular opensearch:opensearch-peers opensearch:opensearch-peers opensearch_peers peer opensearch:upgrade-version-a opensearch:upgrade-version-a upgrade peer self-signed-certificates:certificates opensearch:certificates tls-certificates regular ```default Notice the new relation `opensearch:opensearch-client data-integrator:opensearch opensearch_client regular`. This relation created the user, password, and CA certificate for the `data-integrator` application. To retrieve information such as the username, password, and database. Run the following command: ```bash juju run data-integrator/leader get-credentials ``` This should output something like: ```yaml Running operation 1 with 1 task - task 2 on unit-data-integrator-0 Waiting for task 2... ok: "True" opensearch: data: '{"extra-user-roles": "admin", "index": "test-index", "requested-secrets": "[\"username\", \"password\", \"tls\", \"tls-ca\", \"uris\"]"}' endpoints: 10.95.38.139:9200,10.95.38.212:9200,10.95.38.94:9200 index: test-index password: j3JWFnDkoumCxn0CtKZRCmdRMUlYTZFI tls-ca: |- -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- username: opensearch-client_5 version: 2.14.0 ``` Save the CA certificate (value of `tls-ca` in the previous response), username, and password, because you will need them in the next section. ## Create and access OpenSearch indices Before connecting to OpenSearch, it is mandatory that you [enable TLS on this cluster](https://canonical.com/data/opensearch/docs/2//tutorial/3-enable-encryption.md#tutorial-3-enable-encryption), following the previous step in the tutorial. You can access the OpenSearch REST API in any way you prefer. In this tutorial, we will use `curl`. Get the IP of an OpenSearch node from the output of `juju status` (any of the nodes should work fine), and store the CA certificate in a local file (e.g. `demo-ca.pem`). Then, replacing ``, ``, and `` with the values from the `get-credentials` output above, run the following command to connect to the OpenSearch cluster: ```bash curl --cacert demo-ca.pem -XGET https://:@:9200/ ``` Sending a `GET` request to this `/` endpoint should return some basic information about your OpenSearch deployment, which should look something like this: ```json { "name" : "opensearch-2.0f3", "cluster_name" : "opensearch-x3y6", "cluster_uuid" : "yFS58g6hTbS0VxzJi0u7_g", "version" : { "distribution" : "opensearch", "number" : "2.14.0", "build_type" : "tar", "build_hash" : "30dd870855093c9dca23fc6f8cfd5c0d7c83127d", "build_date" : "2024-05-27T21:17:37.476666822Z", "build_snapshot" : false, "lucene_version" : "9.10.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } ``` If this command fails, ensure the OpenSearch units are all in an active-idle state. The command we ran used the `--cacert` flag to pass in the CA chain generated by the TLS operator, ensuring secure and encrypted HTTP communications between our local host and the OpenSearch node. To recap, the CA chain is generated by the TLS operator. It is passed over to the OpenSearch charm, which provides this certificate in its databag to any application that relates to it using the `opensearch-client` relation interface. When developing charms that relate to the OpenSearch operator, make sure to use this certificate and the user credentials (username/password pair) to authenticate communications. To index some data, run the following command: ```bash curl --cacert demo-ca.pem \ -XPOST https://:@:9200/albums/_doc/1?refresh=true \ -d '{"artist": "Vulfpeck", "genre": ["Funk", "Jazz"], "title": "Thrill of the Arts"}' \ -H 'Content-Type: application/json' -H 'Accept: application/json' ``` This command uses the same certificate and user credentials to send a `POST` request to the same node as before, but it sends a specific JSON payload to a specific document address. The output should look like the following: ```json { "_index": "albums", "_id": "1", "_version": 1, "result": "created", "forced_refresh": true, "_shards": { "total": 2, "successful": 1, "failed": 0 }, "_seq_no": 0, "_primary_term": 1 } ``` Note from the response that our request was successful and the document was indexed. Use the following command to retrieve the previous document: ```bash curl --cacert demo-ca.pem -XGET https://:@:9200/albums/_doc/1 ``` This call should output something like the following: ```json { "_index": "albums", "_id": "1", "_version": 1, "_seq_no": 0, "_primary_term": 1, "found": true, "_source": { "artist": "Vulfpeck", "genre": ["Funk", "Jazz"], "title": "Thrill of the Arts" } } ``` To add data in bulk using the [OpenSearch Bulk API](https://opensearch.org/docs/2.19/api-reference/document-apis/bulk/), copy and paste the following data into a file called `bulk-albums.json`, ensuring that you keep the newline at the end of the file: ```json { "index" : { "_index": "albums", "_id" : "2" } } {"artist": "Herbie Hancock", "genre": ["Jazz"], "title": "Head Hunters"} { "index" : { "_index": "albums", "_id" : "3" } } {"artist": "Lydian Collective", "genre": ["Jazz"], "title": "Adventure"} { "index" : { "_index": "albums", "_id" : "4" } } {"artist": "Rush", "genre": ["Prog"], "title": "Moving Pictures"} ``` Then, to send this data to the bulk endpoint, run the following command: ```bash curl --cacert demo-ca.pem -XPOST https://:@:9200/_bulk --data-binary @bulk-albums.json -H 'Content-Type: application/json' ``` This should return a JSON response with the results of the bulk indexing operation: ```text { "took": 17, "errors": false, "items": [ ... ] } ``` What is of interest in the response is the `errors` field. If it is `false`, then the operation was successful. If it is `true`, then there were errors in the bulk operation. The `items` field contains the results of each operation in the bulk request. To view the previously indexed documents, we can run a search query for the `Jazz` keyword in our `albums` index, using the following command: ```bash curl --cacert demo-ca.pem -XGET https://:@:9200/albums/_search?q=Jazz ``` This should return a JSON response with all the Jazz albums in the index: ```json { "took": 35, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 3, "relation": "eq" }, "max_score": 0.4121628, "hits": [ { "_index": "albums", "_id": "1", "_score": 0.4121628, "_source": { "artist": "Vulfpeck", "genre": ["Funk", "Jazz"], "title": "Thrill of the Arts" } }, { "_index": "albums", "_id": "2", "_score": 0.4121628, "_source": { "artist": "Herbie Hancock", "genre": ["Jazz"], "title": "Head Hunters" } }, { "_index": "albums", "_id": "3", "_score": 0.4121628, "_source": { "artist": "Lydian Collective", "genre": ["Jazz"], "title": "Adventure" } } ] } } ``` ## Remove the user To remove the user used in the previous calls, remove the relation. Removing the relation automatically removes the user created when the relation was created. Run the following to remove the relation: ```shell juju remove-relation opensearch data-integrator ``` if you run `juju status --relations` you will see that the relation has been removed and that the `data-integrator` application is now in a `blocked` state. Model Controller Cloud/Region Version SLA Timestamp tutorial opensearch-demo localhost/localhost 3.6.23 unsupported 13:48:08Z App Version Status Scale Charm Channel Rev Exposed Message data-integrator blocked 1 data-integrator latest/stable 362 no Please relate the data-integrator with the desired product opensearch active 3 opensearch 2/stable 344 no
\\\\ self-signed-certificates active 1 self-signed-certificates 1/stable 586 no Unit Workload Agent Machine Public address Ports Message data-integrator/0\* blocked idle 4 10.95.38.22 Please relate the data-integrator with the desired product opensearch/0 active idle 0 10.95.38.94 9200/tcp
\\\\ opensearch/1 active idle 1 10.95.38.139 9200/tcp
\\\\ opensearch/2\* active idle 2 10.95.38.212 9200/tcp
\\\\ self-signed-certificates/0\* active idle 3 10.95.38.54 Machine State Address Inst id Base AZ Message 0 started 10.95.38.94 juju-be3883-0 ubuntu@24.04 Running 1 started 10.95.38.139 juju-be3883-1 ubuntu@24.04 Running 2 started 10.95.38.212 juju-be3883-2 ubuntu@24.04 Running 3 started 10.95.38.54 juju-be3883-3 ubuntu@24.04 Running 4 started 10.95.38.22 juju-be3883-4 ubuntu@24.04 Running Integration provider Requirer Interface Type Message data-integrator:data-integrator-peers data-integrator:data-integrator-peers data-integrator-peers peer opensearch:node-lock-fallback opensearch:node-lock-fallback node_lock_fallback peer opensearch:opensearch-peers opensearch:opensearch-peers opensearch_peers peer opensearch:upgrade-version-a opensearch:upgrade-version-a upgrade peer self-signed-certificates:certificates opensearch:certificates tls-certificates regular ```default Now try again to connect in the same way as the previous section ```bash curl --cacert demo-ca.pem -XGET https://:@:9200/ ``` This should output something like the following error: ```text Unauthorized ``` If you wanted to recreate this user all you would need to do is relate the two applications and run the same action on `data-integrator` to get the new credentials: ```shell juju integrate data-integrator opensearch ``` Once the relation is active, retrieve the new credentials: ```shell juju run data-integrator/leader get-credentials ``` You can now connect to the database with this new username and password: ```bash curl --cacert demo-ca.pem -XGET https://new_username:new_password@opensearch_node_ip:9200/albums/_search?q=Jazz ``` This should return the same response as before. The data in the database is not lost when the user is removed, only the user is removed. Also, note that the certificate does not change across relations. To create a new certificate, remove the relation between `opensearch` and the `self-signed-certificates` operator, wait for `opensearch` to enter a blocked status, and then recreate the relation. Run the `get-credentials` action on the data-integrator charm again to get the new credentials, and test them again with the above search request. #### NOTE **Next step**: [5. Manage passwords](https://canonical.com/data/opensearch/docs/2//tutorial/5-manage-passwords.md#tutorial-5-manage-passwords) # index.html.md # 6. Scale horizontally > [Charmed OpenSearch Tutorial](https://canonical.com/data/opensearch/docs/2//tutorial/index.md#tutorial-index) > 6. Scale horizontally After having indexed some data in our previous section, let’s take a look at the status of our charm: ```shell juju status ``` The output should look similar to the following: ```text Model Controller Cloud/Region Version SLA Timestamp tutorial opensearch-demo localhost/localhost 3.6.23 unsupported 13:57:38Z App Version Status Scale Charm Channel Rev Exposed Message data-integrator active 1 data-integrator latest/stable 362 no opensearch active 3 opensearch 2/stable 344 no self-signed-certificates active 1 self-signed-certificates 1/stable 586 no Unit Workload Agent Machine Public address Ports Message data-integrator/0* active idle 4 10.95.38.22 opensearch/0 active idle 0 10.95.38.94 9200/tcp opensearch/1 active idle 1 10.95.38.139 9200/tcp opensearch/2* active idle 2 10.95.38.212 9200/tcp self-signed-certificates/0* active idle 3 10.95.38.54 Machine State Address Inst id Base AZ Message 0 started 10.95.38.94 juju-be3883-0 ubuntu@24.04 Running 1 started 10.95.38.139 juju-be3883-1 ubuntu@24.04 Running 2 started 10.95.38.212 juju-be3883-2 ubuntu@24.04 Running 3 started 10.95.38.54 juju-be3883-3 ubuntu@24.04 Running 4 started 10.95.38.22 juju-be3883-4 ubuntu@24.04 Running Integration provider Requirer Interface Type Message data-integrator:data-integrator-peers data-integrator:data-integrator-peers data-integrator-peers peer opensearch:node-lock-fallback opensearch:node-lock-fallback node_lock_fallback peer opensearch:opensearch-client data-integrator:opensearch opensearch_client regular opensearch:opensearch-peers opensearch:opensearch-peers opensearch_peers peer opensearch:upgrade-version-a opensearch:upgrade-version-a upgrade peer self-signed-certificates:certificates opensearch:certificates tls-certificates regular ``` ## Add node You can add an additional node to your deployed OpenSearch application with the following command: ```shell juju add-unit opensearch -n 1 ``` Where `-n 1` specifies the number of units to add. In this case, we are adding one unit to the OpenSearch application. You can add more units by changing the number after `-n`. You can now watch the new units join the cluster with: `juju status --watch 1s`. It usually takes a few minutes for the new nodes to be added to the cluster formation. You’ll know that all four nodes are ready when `juju status --watch 1s` reports: ```text Model Controller Cloud/Region Version SLA Timestamp tutorial opensearch-demo localhost/localhost 3.6.23 unsupported 14:02:18Z App Version Status Scale Charm Channel Rev Exposed Message data-integrator active 1 data-integrator latest/stable 362 no opensearch active 4 opensearch 2/stable 344 no self-signed-certificates active 1 self-signed-certificates 1/stable 586 no Unit Workload Agent Machine Public address Ports Message data-integrator/0* active idle 4 10.95.38.22 opensearch/0 active idle 0 10.95.38.94 9200/tcp opensearch/1 active idle 1 10.95.38.139 9200/tcp opensearch/2* active idle 2 10.95.38.212 9200/tcp opensearch/3 active idle 5 10.95.38.39 9200/tcp self-signed-certificates/0* active idle 3 10.95.38.54 Machine State Address Inst id Base AZ Message 0 started 10.95.38.94 juju-be3883-0 ubuntu@24.04 Running 1 started 10.95.38.139 juju-be3883-1 ubuntu@24.04 Running 2 started 10.95.38.212 juju-be3883-2 ubuntu@24.04 Running 3 started 10.95.38.54 juju-be3883-3 ubuntu@24.04 Running 4 started 10.95.38.22 juju-be3883-4 ubuntu@24.04 Running 5 started 10.95.38.39 juju-be3883-5 ubuntu@24.04 Running ``` You can trust that Charmed OpenSearch added these nodes correctly, and that your replica shards are all assigned. But if you want to verify that your data is correctly replicated, you can also query the shards with the following command: ```bash curl --cacert demo-ca.pem -XGET https://username:password@opensearch_node_ip:9200/_cat/shards ``` Which should result in the following output: ```text test-index 0 r STARTED 0 208b 10.95.38.94 opensearch-0.0f3 test-index 0 p STARTED 0 208b 10.95.38.139 opensearch-1.0f3 .plugins-ml-config 0 r STARTED 1 3.9kb 10.95.38.94 opensearch-0.0f3 .plugins-ml-config 0 r STARTED 1 3.9kb 10.95.38.139 opensearch-1.0f3 .plugins-ml-config 0 p STARTED 1 3.9kb 10.95.38.212 opensearch-2.0f3 .opensearch-observability 0 r STARTED 0 208b 10.95.38.94 opensearch-0.0f3 .opensearch-observability 0 p STARTED 0 208b 10.95.38.139 opensearch-1.0f3 .opensearch-observability 0 r STARTED 0 208b 10.95.38.212 opensearch-2.0f3 albums 0 r STARTED 4 10.7kb 10.95.38.139 opensearch-1.0f3 albums 0 p STARTED 4 10.7kb 10.95.38.212 opensearch-2.0f3 .opensearch-sap-log-types-config 0 r STARTED 10.95.38.94 opensearch-0.0f3 .opensearch-sap-log-types-config 0 r STARTED 10.95.38.139 opensearch-1.0f3 .opensearch-sap-log-types-config 0 p STARTED 10.95.38.212 opensearch-2.0f3 .opendistro_security 0 r STARTED 10 54.2kb 10.95.38.94 opensearch-0.0f3 .opendistro_security 0 r STARTED 10 54.2kb 10.95.38.139 opensearch-1.0f3 .opendistro_security 0 p STARTED 10 155.1kb 10.95.38.212 opensearch-2.0f3 .charm_node_lock 0 r STARTED 1 3.8kb 10.95.38.94 opensearch-0.0f3 .charm_node_lock 0 r STARTED 1 6.9kb 10.95.38.139 opensearch-1.0f3 .charm_node_lock 0 p STARTED 1 11.8kb 10.95.38.212 opensearch-2.0f3 ``` Notice that the shards are distributed across all nodes. ## Remove nodes #### NOTE Refer to [safe-horizontal-scaling guide](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally) to understand how to safely remove units in a production environment. #### WARNING In highly available deployments, only scaling down to 3 nodes is safe. If only 2 nodes are online, neither can be unavailable nor removed. The service will become **unavailable** and **data may be lost** if scaling below 2 nodes. Removing a unit from the Juju application scales down your OpenSearch cluster by one node. Before we scale down the nodes we no longer need, list all the units with `juju status`. Here you will see four units / nodes: `opensearch/0`, `opensearch/1`, `opensearch/2`, `opensearch/3`. To remove the unit `opensearch/3` run: ```bash juju remove-unit opensearch/3 ``` You’ll know that the node was successfully removed when `juju status --watch 1s` reports: ```text Model Controller Cloud/Region Version SLA Timestamp tutorial opensearch-demo localhost/localhost 3.6.23 unsupported 14:05:58Z App Version Status Scale Charm Channel Rev Exposed Message data-integrator active 1 data-integrator latest/stable 362 no opensearch active 3 opensearch 2/stable 344 no self-signed-certificates active 1 self-signed-certificates 1/stable 586 no Unit Workload Agent Machine Public address Ports Message data-integrator/0* active idle 4 10.95.38.22 opensearch/0 active idle 0 10.95.38.94 9200/tcp opensearch/1 active idle 1 10.95.38.139 9200/tcp opensearch/2* active idle 2 10.95.38.212 9200/tcp self-signed-certificates/0* active idle 3 10.95.38.54 Machine State Address Inst id Base AZ Message 0 started 10.95.38.94 juju-be3883-0 ubuntu@24.04 Running 1 started 10.95.38.139 juju-be3883-1 ubuntu@24.04 Running 2 started 10.95.38.212 juju-be3883-2 ubuntu@24.04 Running 3 started 10.95.38.54 juju-be3883-3 ubuntu@24.04 Running 4 started 10.95.38.22 juju-be3883-4 ubuntu@24.04 Running ``` #### NOTE **Next step**: [7. Clean up the environment](https://canonical.com/data/opensearch/docs/2//tutorial/7-clean-up-the-environment.md#tutorial-7-clean-up-the-environment). # index.html.md # Explanation The pages in this section aim to provide additional context and deeper understanding of foundational topics and concepts relevant to OpenSearch. ## Cluster architecture - [Node roles and cluster topology](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles) — how node roles, data tiers, and the main orchestrator pattern work in large deployments. - [Cluster health and scaling](https://canonical.com/data/opensearch/docs/2//explanation/cluster-health.md#explanation-cluster-health) — understanding green/yellow/red health states and safe scaling practices. - [Performance profiles](https://canonical.com/data/opensearch/docs/2//explanation/performance-profiles.md#explanation-performance-profiles) — how profiles tune JVM heap, node count, and resource requirements. - [Persistent storage and disk recovery](https://canonical.com/data/opensearch/docs/2//explanation/persistent-storage.md#explanation-persistent-storage) — risks of disk reuse, dangling indices, and last-resort disaster recovery. ## Security Secure deployments of Charmed OpenSearch can be achieved through using recommended configurations, including setting up encryption and authentication. For more details, see the [Security](https://canonical.com/data/opensearch/docs/2//explanation/security/index.md#explanation-security-index) topic overview, [Cryptography](https://canonical.com/data/opensearch/docs/2//explanation/security/cryptography.md#explanation-security-cryptography), and [TLS certificates](https://canonical.com/data/opensearch/docs/2//explanation/security/tls-certificates.md#explanation-tls-certificates) explanation pages. ## Testing and monitoring For a deeper understanding of monitoring in Charmed OpenSearch, see the [Monitoring](https://canonical.com/data/opensearch/docs/2//explanation/monitoring.md#explanation-monitoring) explanation page. Overview of our approach to testing Charmed OpenSearch is summarized in the [Software testing for charms](https://canonical.com/data/opensearch/docs/2//explanation/software-testing.md#explanation-software-testing) page. # index.html.md # Persistent storage and disk recovery Charmed OpenSearch stores data and cluster metadata on Juju-managed storage volumes. In some scenarios — such as disaster recovery, cluster migration, or hardware replacement — you may need to reuse disks that already contain data from a previous OpenSearch cluster. This page explains the concepts behind disk reuse, the risks involved, and the mechanisms OpenSearch provides for detecting and recovering existing data. ## Why disk reuse is risky When a disk that previously belonged to an OpenSearch cluster is attached to a new node, the disk contains not only indexed data but also **cluster metadata** — the cluster UUID, node identity, and references to peer nodes. If this metadata does not match the current cluster, the node may fail to start or may behave unpredictably. #### CAUTION Reusing disks may cause older data to override existing or newer data. Make sure the disks and their contents are known before proceeding. ## Prefer snapshot and restore Moving data between clusters by reattaching disks is **not** a general-purpose migration path. The only supported, data-safe way to move data between clusters is to [create a backup](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-create-a-backup) of the source cluster and [restore or migrate](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-migrate-a-cluster) it into the target cluster. Reusing a disk from a different cluster involves editing the node’s on-disk **coordination metadata** — the cluster UUID and voting configuration used to elect a cluster manager. This is a last resort for disaster recovery, valid only when the source cluster is permanently gone (for example, after losing a majority of [`cluster_manager`-eligible](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles-available) nodes, or after a brutal cluster decommission) **and** no viable snapshot exists to restore from instead. ## How OpenSearch detects existing data OpenSearch provides two mechanisms for interacting with data on an attached disk: ### The `/_dangling` API When a node starts with a disk that contains index data not known to the current cluster, those indices are called **dangling indices**. The [dangling indices API](https://opensearch.org/docs/2.19/api-reference/index-apis/dangling-index/) allows you to list, import, or delete them. #### CAUTION The dangling indices API cannot guarantee that the imported data accurately represents the latest state of the data when the index was still part of the original cluster. ### The `opensearch-node` CLI The `opensearch-node` command-line tool performs **coordination-metadata surgery** on a stopped node’s disk: it edits the cluster UUID and voting configuration directly on disk, bypassing the normal cluster-formation and consensus protocol. It is **not** a general-purpose way to migrate disks between clusters. #### CAUTION `detach-cluster` and `unsafe-bootstrap` are last-resort disaster-recovery commands. OpenSearch warns that they can cause **arbitrary data loss**, because the node running the command may not hold the most recent cluster metadata. Only use them after the **permanent** loss of a majority (or all) of the [`cluster_manager`-eligible](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles-available) nodes in a cluster, or after a brutal cluster decommission, and only when no viable snapshot recovery exists. A success message from either command does not mean no data was lost — always audit the data after recovery. Two subcommands are relevant: - **`unsafe-bootstrap`** — use only when half or more of the `cluster_manager`-eligible nodes are permanently lost, the cluster can no longer form a quorum, and snapshot recovery is not possible. The command creates a new cluster UUID and bootstraps a new cluster from one surviving eligible node’s persisted metadata. If multiple eligible nodes survived, choose the one with the highest cluster-state (term, version) pair. - **`detach-cluster`** — use to detach a node from its old cluster (by resetting cluster UUID), so it can join a replacement cluster. Do this only when snapshot recovery is impossible and either all `cluster_manager`-eligible nodes have been permanently lost or a replacement cluster has already been created with `unsafe-bootstrap` from an eligible node in the same original cluster. ## Scenarios for disk reuse There are three scenarios, each requiring a different approach: ### Same cluster — reattaching a detached volume If a volume was previously used by a node in the same cluster and was detached (e.g. due to a unit removal), it can be reattached to a new unit. The node rejoins the cluster automatically with no metadata changes required — the cluster UUID and peer references already match. ### Different cluster — attaching to an existing cluster (last resort) When a disk from a different cluster is attached to a new unit in an existing cluster, the node holds stale metadata referencing the old cluster UUID. The node will fail to join because it cannot reach its old peers. This is only appropriate if the source cluster is permanently gone and no viable snapshot exists to restore from instead — in that case, the last-resort `detach-cluster` tool can be run to discard the stale metadata so the node can join the new cluster. ### Different cluster — bootstrapping a new cluster (last resort) When a disk from a different cluster is used to seed a brand new single-node deployment, the node will fail to start because it is trying to load its original metadata and cannot reach any of its old peers. Again, this only makes sense if the source cluster is permanently gone and no viable snapshot exists — in that case, the last-resort `unsafe-bootstrap` tool can be run to reset the cluster metadata, after which a new cluster UUID is assigned and the node starts fresh. ## See also - [How to back up and restore](https://canonical.com/data/opensearch/docs/2//how-to/back-up-and-restore/index.md#how-to-guides-back-up-and-restore-index) — the preferred, data-safe way to move data between clusters. - [How to manage persistent storage](https://canonical.com/data/opensearch/docs/2//how-to/persistent-storage.md#how-to-persistent-storage) — step-by-step guide, including the last-resort disaster-recovery procedure. - [Node roles and cluster topology](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles) — how nodes and clusters are structured. # index.html.md # Node roles and cluster topology OpenSearch clusters are made up of nodes, each assigned one or more **roles** that determine what work the node performs. In Charmed OpenSearch, roles are assigned at the Juju **application** level — all units within an application share the same set of roles. This page explains the available roles, how they interact, and how multiple applications combine to form a large deployment. ## Available node roles Charmed OpenSearch accepts the following comma-separated values in the `roles` configuration option: | Role | Description | |----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `cluster_manager` | Handles cluster-wide operations: creating and deleting indices, managing shard allocation, and rebalancing data. One node is elected **cluster manager** among all `cluster_manager`-eligible nodes. | | `data` | Stores indexed data and performs search and indexing operations. Data nodes hold the shards that contain the indexed documents. | | `data.` | The `data` role, with a data tier assigned. The temperature is one of `hot` (most recent, most frequently queried data), `warm` (queried less frequently but still searchable), `cold` (infrequently accessed), `frozen`, or `content`. | | `ingest` | Pre-processes documents before they are indexed (pipelines, transformations). | | `coordinating` | Routes requests to the appropriate data nodes and aggregates results. Does not hold data. | | `voting_only` | Participates in cluster manager election but is never eligible to become the cluster manager node itself. | | `ml` | Runs machine learning tasks such as model training and inference. | #### NOTE The `data.` values are a Charmed OpenSearch convenience, not OpenSearch roles. Upstream, the role is `data` and the tier is a separate node attribute. When you set `data.hot`, the charm assigns the `data` role and sets `node.attr.temp` to `hot`. Only one temperature may be set per application. Classifying data nodes into **tiers** supports [index lifecycle management](https://opensearch.org/docs/2.19/im-plugin/ism/index/) policies that move data between tiers as it ages. Note that the charm only tags nodes with their temperature — it does not place data on them. To route indices to the appropriate tier, you need to configure the index lifecycle management policies yourself. ## Auto-generated roles When the `roles` configuration option is left empty, the charm automatically assigns the following roles to all nodes in the application: ```python ["data", "ingest", "ml", "cluster_manager"] ``` This means a single-application deployment can handle all functions — cluster management, data storage, ingestion, and machine learning — without any additional configuration. ## Setting roles Roles can be set at deployment time or changed later via configuration: ```shell # At deployment time juju deploy opensearch -n 3 --config roles="cluster_manager,data,ml" # After deployment (triggers a rolling restart) juju config opensearch roles="cluster_manager,data,ml" ``` #### NOTE Removal of the `cluster_manager` role is never supported: the charm rejects the change and the application becomes `blocked`. Removal of the `data` role is rejected in a simple (single-application) deployment, because the data on disk could no longer be served. In a [large deployment](#explanation-node-roles-large), it is allowed only if another application in the cluster still holds the `data` role, so that shards can be reallocated. ## Large deployments For production workloads, a single application is often insufficient. Charmed OpenSearch supports **large deployments** — a single OpenSearch cluster composed of multiple Juju applications, each configured with specific node roles. This topology allows you to scale different node types independently and provides fault tolerance. ### The main orchestrator pattern A large deployment consists of three types of applications: 1. **Main orchestrator** — the primary application that bootstraps the cluster. It has `init_hold=false` (the default) and is responsible for generating the cluster UUID, initialising the security index, and sharing admin certificates with other applications. When no roles are explicitly set, it receives all auto-generated roles. 2. **Failover orchestrator** (recommended) — a dedicated `cluster_manager` application that takes over orchestration if the main orchestrator fails or is removed. It must have `init_hold=true` to prevent it from starting before being integrated with the main orchestrator. 3. **Data applications** — applications with `data`-tier roles (e.g. `data.hot`, `data.warm`) that store and process data. They also require `init_hold=true`. ### Critical configuration rules Two configuration options are essential for large deployments: - **`cluster_name`**: All applications must share the same `cluster_name` value. A mismatch prevents applications from forming a cluster. If left unset on the main orchestrator, a name is auto-generated and inherited by other applications via the peer-cluster relation. - **`init_hold`**: Must be set to `true` on every application **except** the main orchestrator. This prevents non-orchestrator applications from starting before they are integrated with the main orchestrator, which would cause them to fail (they cannot reach the cluster to obtain admin certificates and cluster metadata). ### Cluster formation The applications are connected via the **peer-cluster** relations: ```shell juju integrate main:peer-cluster-orchestrator failover:peer-cluster juju integrate main:peer-cluster-orchestrator data-hot:peer-cluster juju integrate failover:peer-cluster-orchestrator data-hot:peer-cluster ``` Once these relations are established, the main orchestrator shares the cluster UUID, admin certificates, and security configuration with the other applications. The other applications then start and join the cluster. ### Integrations and the main orchestrator In a large deployment, integrations with external charms (e.g. SMTP, JWT, OAuth) must target the **main orchestrator** application, not the data or failover applications. The main orchestrator is responsible for distributing configuration to the rest of the cluster. If integrated with the wrong application, the charm will enter a `blocked` state. You can identify the main orchestrator by inspecting the integrations section of `juju status --relations` — look for the application providing the `peer-cluster-orchestrator` endpoint. ## See also - [How to launch a large deployment](https://canonical.com/data/opensearch/docs/2//how-to/deploy/large.md#how-to-deploy-large) — step-by-step deployment guide. - [Performance profiles](https://canonical.com/data/opensearch/docs/2//explanation/performance-profiles.md#explanation-performance-profiles) — resource tuning for different deployment sizes. - [Cluster health and scaling](https://canonical.com/data/opensearch/docs/2//explanation/cluster-health.md#explanation-cluster-health) — understanding cluster health states. # index.html.md # Monitoring The OpenSearch charm integrates with the Canonical Observability Stack (COS) to provide infrastructure and cluster health monitoring through metrics, dashboards, alert rules, and logs. COS uses Prometheus for metrics collection, Grafana for visualization, Loki for log aggregation, and Alertmanager for alerting. This enables monitoring of OpenSearch cluster performance, resource utilization (CPU, memory, disk I/O), cluster statistics (node health, shard allocation, indexing rates), and operational health. The way telemetry reaches COS depends on the charm variant: - On **VMs**, the `opensearch` machine charm exposes a `cos-agent` endpoint, and a machine `grafana-agent` charm collects the telemetry and forwards it to the COS applications (which typically run in a separate Kubernetes model). - On **Kubernetes**, the `opensearch-k8s` charm exposes native COS endpoints (`metrics-endpoint`, `grafana-dashboard`, and `logging`), and a `grafana-agent-k8s` charm in the OpenSearch model collects the telemetry and forwards it to the COS applications. When COS runs in the same model, OpenSearch can integrate directly with the Prometheus, Grafana, and Loki applications instead. #### NOTE See: [How to enable monitoring](https://canonical.com/data/opensearch/docs/2//how-to/monitoring.md#how-to-monitoring) via COS and Grafana. Additionally, Charmed OpenSearch can integrate with the [OpenSearch Dashboards](https://canonical-charmed-opensearch-dashboards.readthedocs-hosted.com/2/) charm for exploring and visualizing your indexed business or application data. While COS with Grafana monitors your OpenSearch infrastructure and operational health, OpenSearch Dashboards provides purpose-built tools for interactive data exploration, custom visualizations, query builders, and OpenSearch-specific features like anomaly detection. Most production deployments benefit from using both: COS for infrastructure monitoring and OpenSearch Dashboards for data analysis. #### NOTE See: [How to deploy OpenSearch Dashboards](https://canonical-charmed-opensearch-dashboards.readthedocs-hosted.com/2/how-to/deploy/) charm. ## Metrics The charm enables the Prometheus Exporter plugin for OpenSearch by default: [The Prometheus Exporter Plugin for OpenSearch](https://github.com/Aiven-Open/prometheus-exporter-plugin-for-opensearch) The meaning of the metrics collected can be found in the upstream documentation: - [indices_stats_metrics](https://opensearch.org/docs/2.19/api-reference/index-apis/stats/) - [nodes_stats_metrics](https://opensearch.org/docs/2.19/api-reference/nodes-apis/nodes-stats/) - [cluster_stats_metrics](https://opensearch.org/docs/2.19/api-reference/cluster-api/cluster-stats/) ## Alert rules The charm deploys a pre-configured set of Prometheus alert rules by default. To ensure you are referencing the latest default alert rules, check the source file of alert definitions in the repository: - VM: [machine prometheus_alerts.yaml](https://github.com/canonical/opensearch-operator/blob/2/edge/machine/src/alert_rules/prometheus/prometheus_alerts.yaml) - K8s: [kubernetes prometheus_alerts.yaml](https://github.com/canonical/opensearch-operator/blob/2/edge/kubernetes/src/alert_rules/prometheus/prometheus_alerts.yaml) The K8s charm additionally ships a set of Loki log-based alert rules, see [opensearch.rules](https://github.com/canonical/opensearch-operator/blob/2/edge/kubernetes/src/loki_alert_rules/opensearch.rules). ## Logs All the logs from the OpenSearch payload are available in the Grafana GUI at `Home > Explore` To get OpenSearch logs, go to the `Label filters` field and set `juju_application` to the name of your OpenSearch application (for example, `opensearch` on VMs or `opensearch-k8s` on Kubernetes), select one operation, e.g. `Line contains` and run the query. The following Grafana Explore screenshot shows OpenSearch logs from a VM deployment: ![Grafana Explore showing OpenSearch logs filtered by Juju application](how-to/img/cos-1.png) ## Grafana dashboard The **Charmed OpenSearch** Grafana dashboard provides an at-a-glance view of cluster health, node resource utilization, and indexing throughput. The dashboard includes panels for CPU, memory, disk I/O, JVM heap, shard counts, and document indexing rates. You can filter the displayed data using the selectors at the top of the dashboard: - **Juju model** — the Juju model the cluster is deployed in - **Juju application** — the Juju application (e.g. `opensearch` or `opensearch-main`) - **Juju unit** — an individual Juju unit within the application - **Cluster** — the OpenSearch cluster name (useful when multiple clusters share a COS instance) - **Role** — filter by OpenSearch node role (e.g. `cluster_manager`, `data`) ![Charmed OpenSearch Grafana dashboard — overview panel](how-to/img/dash1.png) ![Charmed OpenSearch Grafana dashboard — node detail panel](how-to/img/dash2.png) # index.html.md # Cluster health and scaling OpenSearch reports cluster health as a single status that summarises the state of shard allocation across all nodes. Charmed OpenSearch reflects this health in the Juju application status, making it visible without querying the OpenSearch API directly. Understanding these health states is essential for safe scaling operations — particularly when removing nodes. ## Cluster health states The OpenSearch [cluster health API](https://opensearch.org/docs/2.19/api-reference/cluster-api/cluster-health/) returns one of three states: | State | Meaning | Scaling down safe? | |--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------| | **`green`** | All primary and replica shards are allocated. The cluster is fully healthy. | Likely safe — but verify the target node does not hold a primary shard of an unreplicated index. | | **`yellow`** | All primary shards are allocated, but some replica shards are not. The cluster is functional but lacks full redundancy. This is either **temporary** (replicas are still initializing or relocating) or **permanent** (replicas cannot be assigned at all). | **May not be safe** — removing a node could lose the only copy of a primary shard if replicas are unavailable. | | **`red`** | At least one primary shard is **unassigned**. Some data is currently unavailable. | **Not safe** — do not remove nodes. [Diagnose the cause]() and restore health first. | ### Temporary and permanent `yellow` Only one of the two `yellow` cases needs your intervention: - **Temporary** — shards are `initializing` or `relocating`. Normal after adding or removing a unit; resolves on its own. - **Permanent** — shards are `unassigned` and cannot be allocated, typically because there are too few nodes or an allocation setting prevents their placement. Does not resolve on its own; usually requires [scaling up](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally). Check the number of shards in each state in the [cluster health API](https://opensearch.org/docs/2.19/api-reference/cluster-api/cluster-health/) response to tell them apart, and see the matching [alert rule](https://canonical.com/data/opensearch/docs/2//reference/alert-rules.md#ref-alert-rules) for each: a non-zero `initializing_shards` or `relocating_shards` (`OpenSearchClusterYellowTemp`) is temporary, while a non-decreasing `unassigned_shards` (`OpenSearchClusterYellow`) is permanent. To find out why an unassigned shard cannot be allocated, use the [cluster allocation explain API](https://opensearch.org/docs/2.19/api-reference/cluster-api/cluster-allocation/). ### Why a cluster turns `red` A `red` status means at least one primary shard is unassigned, so the data in that shard cannot be read or written. It does **not**, by itself, mean the data is permanently lost. There are three broad causes, and they call for different responses: - **Insufficient capacity** — there is too little disk space, or too few nodes suitable to hold the shard. Adding capacity resolves it. - **Allocation rules** — awareness attributes, filters, or other allocation settings prevent the shard from being placed on any node. The rules themselves must be corrected. - **No valid shard copy** — no available node holds a usable copy of the primary shard. Adding an empty node cannot recreate that data; recovery requires bringing the failed node back, or restoring from a snapshot. This is why scaling up is not a universal remedy for a `red` cluster. Use the [cluster allocation explain API](https://opensearch.org/docs/2.19/api-reference/cluster-api/cluster-allocation/) to establish which cause applies before acting — a `no_valid_shard_copy` decision points to the third case. But if the cause can be fixed by scaling, see [how to scale a cluster horizontally](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally) for instructions. ### How health maps to Juju status The charm reflects cluster health in the application status: - **`active`** — the cluster is healthy (equivalent to `green`). - **`maintenance`** — shards are still initializing or relocating (a temporary `yellow`). The message is *“Some shards are still initializing / relocating.”* - **`waiting`** — the charm is waiting for an operation to complete, for example *“Waiting for OpenSearch to start…”* or for specific shards to finish building. - **`blocked`** — the cluster has issues (a permanent `yellow`, or `red`). The status message describes the problem: *“1 or more ‘replica’ shards are not assigned, please scale your application up.”* for a permanent `yellow`, and *“1 or more ‘primary’ shards are not assigned, please scale your application up.”* for `red`. This means you can use `juju status` as a quick health check without querying the OpenSearch API directly. The charm derives these statuses on a best-effort basis, so treat the OpenSearch health API as the source of truth when it matters — for example, before [scaling down](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally). ### Mapping Juju units to OpenSearch nodes Each Juju unit runs one OpenSearch node. To confirm a node’s identity from OpenSearch’s side — for example, after adding, removing, or recovering a node — query the [CAT nodes API](https://opensearch.org/docs/2.19/api-reference/cat/cat-nodes/): ```shell curl --cacert cert.pem -XGET "https://:9200/_cat/nodes?v" -u admin: ``` ```text ip ... node.roles cluster_manager name 10.81.173.167 ... cluster_manager,data,ingest,ml * opensearch-1.f1a 10.81.173.48 ... cluster_manager,data,ingest,ml - opensearch-2.f1a ``` `name` and `ip` identify the Juju unit (`opensearch-1.f1a` is `opensearch/1`) and its `Public address`. `node.roles` lists the [assigned roles](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles), and `cluster_manager` marked `*` identifies the currently elected cluster manager. ## Shard allocation To understand why health states matter for scaling, it helps to understand shard allocation: - Each index is divided into **shards**. Each shard has a **primary** copy and zero or more **replica** copies distributed across nodes. - When a node is removed, its shards must be **reallocated** to remaining nodes. If a primary shard’s node is removed and no replica is available, the shard becomes unassigned (health turns `red`). - If replicas are available, OpenSearch promotes a replica to primary and creates new replicas on remaining nodes (health may temporarily drop to `yellow`). You can inspect shard allocation with: ```shell curl --cacert cert.pem -XGET "https://:9200/_cat/shards" -u admin: ``` And explain why a shard is unassigned with the [cluster allocation explain API](https://opensearch.org/docs/2.19/api-reference/cluster-api/cluster-allocation/): ```shell curl --cacert cert.pem -XGET "https://:9200/_cluster/allocation/explain" -u admin: ``` ## Safe scaling practices **Adding nodes** is always safe. New units automatically join the cluster and start receiving shard allocations. No additional configuration is required. **Removing nodes** requires caution: 1. **Check health first** — confirm the cluster is `green` (or at minimum `yellow` with sufficient replicas) before removing any unit. 2. **Remove one unit at a time** — never remove multiple units in a single command. After each removal, wait for the cluster to rebalance and return to `green` before removing the next unit. 3. **Monitor during and after removal** — the charm requests OpenSearch to move shards away from the departing data node to other data nodes before stopping it. If the node participates in cluster-manager voting, the charm also excludes it from the voting configuration. If it is the elected cluster manager, OpenSearch elects a replacement. Wait for the application to fully stabilise before proceeding. #### NOTE In highly available deployments, **do not scale below 3 nodes**. ### Reactive blocking The charm **reactively** (not proactively) blocks unsafe removals. This means it does not know in advance whether removing a specific unit will put the cluster in a `red` state. If health degrades to `red` after a removal, the charm will block further removals to give you the opportunity to recover — usually by scaling back up, though the appropriate action depends on [why the cluster turned `red`](). ## See also - [How to scale a cluster horizontally](https://canonical.com/data/opensearch/docs/2//how-to/scale-horizontally.md#how-to-scale-horizontally) — step-by-step scaling guide. - [Node roles and cluster topology](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles) — how node roles affect scaling. - [Performance profiles](https://canonical.com/data/opensearch/docs/2//explanation/performance-profiles.md#explanation-performance-profiles) — resource requirements for different deployment sizes. # index.html.md # Performance profiles Charmed OpenSearch provides **performance profiles** that tune resource allocation and OpenSearch parameters to suit different use cases. Profiles simplify deployment by automatically configuring JVM heap size, node count requirements, and system-level constraints based on the intended deployment scenario. ## Why profiles exist OpenSearch is sensitive to resource configuration. Setting JVM heap too low causes out-of-memory errors; setting it too high wastes RAM and can cause long garbage collection pauses. Node count affects cluster stability: a single node cannot provide high availability. Profiles encapsulate these best practices so that users do not need to manually tune JVM heap, memory limits, and node constraints. They also enforce safety checks: for example, the `production` profile will block deployment if the minimum node count is not met. ## Available profiles Charmed OpenSearch supports two profiles: | Setting | `testing` | `production` | |--------------------|---------------------------------------------|--------------------------------------------| | Cluster size | Minimum 1 node (cluster manager + data) | Minimum 3 cluster manager + 3 data nodes | | Memory requirement | None enforced | Minimum 8 GB per node | | JVM heap | Fixed at 1 GB | 50% of available RAM (min 4 GB, max 31 GB) | | Use case | Development, testing, lightweight workloads | Production workloads and large deployments | ### The `testing` profile Designed for development and testing on a single host. It allows a cluster to run with a single node and sets the JVM heap to a fixed 1 GB. No strict memory requirements are enforced. A warning is logged indicating that this profile is not suitable for production. ### The `production` profile Designed for production workloads. It enforces: - A minimum of 3 cluster manager nodes and 3 data nodes for high availability. - JVM heap set to 50% of available RAM, capped at 31 GB. Since the profile also requires at least 8 GB of RAM per node, the heap is effectively at least 4 GB. This follows the [OpenSearch recommendation](https://opensearch.org/docs/2.19/tuning-your-cluster/performance/) to allocate roughly half of available RAM to the JVM heap, leaving the rest for the operating system file cache. - A minimum of 8 GB of RAM per node. If any of these requirements are not met, the charm remains in a `blocked` state until corrected. ## System prerequisites Both profiles enforce the same system-level prerequisites: - **Swap must be disabled** — OpenSearch performs best when it is not swapped to disk. - `vm.max_map_count >= 262144` — required by OpenSearch for mmap-based file access. Setting `fs.file-max = 1048576` is strongly recommended to ensure sufficient file descriptors for large deployments, but it is not currently enforced by either profile. The VM charm sets `net.ipv4.tcp_retries2` automatically. The K8s charm does not; for optional pod-level tuning, see [Configure TCP retries](https://canonical.com/data/opensearch/docs/2//how-to/deploy/standard.md#how-to-deploy-tcp-retries). For instructions on how to apply these settings, see [How to deploy](https://canonical.com/data/opensearch/docs/2//how-to/deploy/standard.md#how-to-deploy-standard). For the full list of required kernel parameters, see [System requirements](https://canonical.com/data/opensearch/docs/2//reference/system-requirements.md#reference-system-requirements). ## Changing profiles The profile can be changed after deployment. The charm automatically detects the update and reconfigures the instance to match the new profile’s requirements. ```shell juju config opensearch profile= ``` Where `` is `testing` or `production`. #### NOTE Switching from `testing` to `production` may cause the charm to enter a `blocked` state if the current node count does not meet the production minimum (3 cluster manager + 3 data nodes). Scale up the cluster before switching profiles. ## See also - [How to optimize cluster performance with profiles](https://canonical.com/data/opensearch/docs/2//how-to/optimize-cluster-performance.md#how-to-optimize-cluster-performance) — step-by-step guide. - [System requirements](https://canonical.com/data/opensearch/docs/2//reference/system-requirements.md#reference-system-requirements) — hardware, software, and kernel parameter requirements. - [Node roles and cluster topology](https://canonical.com/data/opensearch/docs/2//explanation/node-roles.md#explanation-node-roles) — how node roles relate to cluster sizing. # index.html.md # Software testing for charms #### NOTE All commands are written for `juju >= v.3.1`. Charmed OpenSearch no longer supports `juju v.2`. Most types of standard [software tests](https://en.wikipedia.org/wiki/Software_testing) are applicable to Charmed OpenSearch. #### NOTE The charm logic lives in the [`opensearch-single-kernel-library`](https://github.com/canonical/opensearch-single-kernel-library), which contains the full unit and integration test suites. The charm repositories only carry minimal smoke-level integration tests that validate the charm wiring. Unless stated otherwise, run the tests described below in the library repository. ## Unit test The unit tests live in the [`opensearch-single-kernel-library`](https://github.com/canonical/opensearch-single-kernel-library) repository. Clone it and run: ```bash tox run -e unit ``` ## Integration test The integration test coverage is rather rich in the OpenSearch charm. Most of it lives in the [`opensearch-single-kernel-library`](https://github.com/canonical/opensearch-single-kernel-library) repository (`tox run -e integration` there); the charm repository itself carries a minimal smoke-level suite that runs in [CI](https://github.com/canonical/opensearch-operator/blob/2/edge/.github/workflows/integration_test.yaml). For high availability (HA) related tests, each test serves as an integration as well as a smoke test with continuous writes routine being perpetually ran in parallel of whatever operation the test is involved in. These continuous writes ensure the availability of the service under different conditions. HA tests make use of one of the 2 fixtures: - `c_writes_runner`: creates an index with a default replication factor and continuously “bulk” feeds data to it - `c_balanced_writes_runner`: creates an index with 2 primary shards and as many replica shards as the number of nodes available in the cluster, and continuously “bulk” feeds data to it. After each test completes, the index gets deleted. ## Performance test Refer to the [OpenSearch VM benchmark](https://canonical.com/data/opensearch/docs/2//how-to/perform-load-testing.md#how-to-perform-load-testing) guide for charmed OpenSearch. # index.html.md # Security hardening guide This document provides an overview of security features and guidance for hardening the security of [Charmed OpenSearch](https://charmhub.io/opensearch) deployments, including setting up and managing a secure environment. ## Environment The environment where Charmed OpenSearch operates can be divided into two components: 1. Cloud 2. Juju ### Cloud Charmed OpenSearch can be deployed on top of several clouds and virtualization layers: | Cloud | Security guides | |-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | OpenStack | [OpenStack Security Guide](https://docs.openstack.org/security-guide/) | | AWS | [Best Practices for Security, Identity and Compliance](https://aws.amazon.com/architecture/security-identity-compliance), [AWS security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html) | | Azure | [Azure security best practices and patterns](https://learn.microsoft.com/en-us/azure/security/fundamentals/best-practices-and-patterns), [Managed identities for Azure resource](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/) | ### Juju Juju is the component responsible for orchestrating the entire lifecycle, from deployment to Day 2 operations. For more information on Juju security hardening, see the [Juju security](https://documentation.ubuntu.com/juju/latest/explanation/juju-security/) page and the [How to harden your deployment](https://documentation.ubuntu.com/juju/3.6/howto/manage-your-juju-deployment/harden-your-juju-deployment) guide. #### Cloud credentials When configuring cloud credentials to be used with Juju, ensure that users have correct permissions to operate at the required level. Juju superusers responsible for bootstrapping and managing controllers require elevated permissions to manage several kinds of resources, such as virtual machines, networks, storage, etc. Please refer to the links below for more information on the policies required to be used depending on the cloud. | Cloud | Cloud user policies | |-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | OpenStack | N/A | | AWS | [Juju AWS Permission](https://discourse.charmhub.io/t/juju-aws-permissions/5307), [AWS Instance Profiles](https://discourse.charmhub.io/t/using-aws-instance-profiles-with-juju-2-9/5185), [Juju on AWS](https://juju.is/docs/juju/amazon-ec2) | | Azure | [Juju Azure Permission](https://juju.is/docs/juju/microsoft-azure), [How to use Juju with Microsoft Azure](https://discourse.charmhub.io/t/how-to-use-juju-with-microsoft-azure/15219) | #### Juju users It is very important that Juju users are set up with minimal permissions depending on the scope of their operations. Please refer to the [User access levels](https://juju.is/docs/juju/user-permissions) documentation for more information on the access levels and corresponding abilities. Juju user credentials must be stored securely and rotated regularly to limit the chances of unauthorized access due to credentials leakage. ## Applications In the following, we provide guidance on how to harden your deployment using: 1. Operating system 2. Security upgrades 3. Encryption 4. Authentication 5. Monitoring ### Operating system Charmed OpenSearch and Charmed OpenSearch Dashboards currently run on top of Ubuntu 22.04 LTS. Deploy a [Landscape Client Charm](https://charmhub.io/landscape-client?) to connect the underlying VM to a Landscape User Account to manage security upgrades and integrate [Ubuntu Pro](https://ubuntu.com/pro) subscriptions. ### Security upgrades Charmed OpenSearch and Charmed OpenSearch Dashboards operators install a pinned revision of the [Charmed OpenSearch snap](https://snapcraft.io/opensearch) and [Charmed OpenSearch Dashboards snap](https://snapcraft.io/opensearch-dashboards), respectively, to provide reproducible and secure environments. New versions of Charmed OpenSearch and Charmed OpenSearch Dashboards may be released to provide patching of vulnerabilities (CVEs). It is important to refresh the charm regularly to make sure the workload is as secure as possible. For more information on how to refresh the charm, see the [how-to upgrade](https://canonical.com/data/opensearch/docs/2//how-to/upgrade.md#how-to-minor-upgrade) guide. ### Encryption Charmed OpenSearch is deployed with encryption enabled. To do that, you need to relate Charmed OpenSearch and Charmed OpenSearch Dashboards to one of the TLS certificate operator charms. Please refer to the [Charming Security page](https://charmhub.io/topics/security-with-x-509-certificates) for more information on how to select the right certificate provider for your use case. For more information on encryption, see the [Cryptography](https://canonical.com/data/opensearch/docs/2//explanation/security/cryptography.md#explanation-security-cryptography) explanation page and the [How to enable encryption](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-enable-tls-encryption) guide. ### Authentication Charmed OpenSearch supports the password [authentication](https://canonical.com/data/opensearch/docs/2//tutorial/5-manage-passwords.md#tutorial-5-manage-passwords). ### Monitoring Charmed OpenSearch provides native integration with the [Canonical Observability Stack (COS)](https://charmhub.io/topics/canonical-observability-stack). To reduce the blast radius of infrastructure disruptions, the general recommendation is to deploy COS and the observed application into separate environments, isolated from one another. Refer to the [COS production deployments best practices](https://charmhub.io/topics/canonical-observability-stack/reference/best-practices) for more information. For instructions, see the [How to integrate the Charmed OpenSearch deployment with COS](https://canonical.com/data/opensearch/docs/2//how-to/monitoring.md#how-to-monitoring) guide. ## Additional resources - [TLS certificates](https://canonical.com/data/opensearch/docs/2//explanation/security/tls-certificates.md#explanation-tls-certificates) — the three certificate categories, rotation mechanics, and CA rotation. - [Cryptography](https://canonical.com/data/opensearch/docs/2//explanation/security/cryptography.md#explanation-security-cryptography) — resource checksums, source verification, encryption, and authentication. - [How to manage TLS encryption](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-enable-tls-encryption) — step-by-step guide for enabling TLS and rotating certificates. For details on the cryptography used by Charmed OpenSearch, see the [Cryptography](https://canonical.com/data/opensearch/docs/2//explanation/security/cryptography.md#explanation-security-cryptography) explanation page. # index.html.md # TLS certificates TLS encryption is mandatory in Charmed OpenSearch — the charm does not function without it. This page explains the three categories of TLS certificates the charm manages, how certificate rotation works, and how CA rotation triggers rolling restarts. For details on the cryptographic implementation (checksums, source verification, keystore and truststore management), see [Cryptography](https://canonical.com/data/opensearch/docs/2//explanation/security/cryptography.md#explanation-security-cryptography). ## Certificate categories Charmed OpenSearch uses three categories of TLS certificates, each serving a different communication channel: | Category | Used for | Can be set on | |------------------|--------------------------------------------------------------------------------------|------------------------------| | `app-admin` | Administrative actions on the OpenSearch cluster (e.g. cluster management API calls) | Leader unit only | | `unit-transport` | Internal communication between OpenSearch nodes (internode traffic) | All units (including leader) | | `unit-http` | External communication between OpenSearch and clients (users or applications) | All units (including leader) | Separating certificates by channel allows independent rotation of each category without affecting the others. For example, you can rotate the `unit-http` certificate (used by external clients) without disrupting internode communication. ## Certificate lifecycle Certificates are managed through the TLS certificate provider charm (e.g. `self-signed-certificates` or `manual-tls-certificates`). The charm generates a Certificate Signing Request (CSR) for each unit and category, sends it to the provider for signing, and stores the returned certificate in a password-protected keystore. ### Automatic rotation on expiry When a certificate expires or is about to expire, the charm automatically requests a new certificate from the provider. No manual intervention is required. ### Manual rotation You can manually trigger certificate rotation by regenerating the private key for the desired category. This generates a new CSR, which is sent to the provider for signing. Once signed, the new certificate is automatically applied to the cluster. ```shell juju run opensearch/leader set-tls-private-key category= ``` Where `` is `app-admin`, `unit-transport`, or `unit-http`. ## CA certificate rotation The CA certificate is provided by the TLS operator and is used to sign all TLS certificates. Rotating the CA certificate is a more involved process because all nodes must trust the new CA before they can communicate. ### With `self-signed-certificates` Trigger CA rotation by changing the common name: ```shell juju config self-signed-certificates ca-common-name= ``` This causes the following sequence: 1. A new CA certificate is generated with the new common name. 2. All previously issued TLS certificates are revoked. 3. OpenSearch detects the revoked certificates and automatically requests new ones. 4. Once new certificates are issued, a **rolling restart** is triggered to apply the new CA across all nodes. During the transition, nodes trust both the old and the new CA: the old CA is kept until the new one is applied to all nodes. This keeps the cluster operational throughout the rolling restart. ### With `manual-tls-certificates` CA rotation with the `manual-tls` operator requires manual signing of CSRs with the new CA certificate. The process is: 1. Sign the existing (or newly generated) CSRs with the new CA certificate. 2. Provide the new certificates to each unit, **starting with the leader**. 3. After receiving the new CA, each node generates new CSRs that must be signed with the new CA and provided back. 4. Repeat for every unit in the cluster. #### CAUTION Always distribute certificates to the leader unit first, then to the remaining nodes. Distributing out of order can cause cluster communication failures. Once all units have the new CA, OpenSearch reloads certificates. A rolling restart is only required if the issuer, subject, or subject alternative names (SANs) of the new certificate differ from the previous one. If they are the same, certificates can be reloaded on the fly without a restart. ## See also - [How to manage TLS encryption](https://canonical.com/data/opensearch/docs/2//how-to/tls-encryption.md#how-to-enable-tls-encryption) — step-by-step guide for enabling TLS, updating keys, and rotating certificates. - [Cryptography](https://canonical.com/data/opensearch/docs/2//explanation/security/cryptography.md#explanation-security-cryptography) — details on encryption, keystore, truststore, and authentication. - [Security hardening guide](https://canonical.com/data/opensearch/docs/2//explanation/security/index.md#explanation-security-index) — overall security guidance. # index.html.md # Cryptography This document describes the cryptography used by Charmed OpenSearch. ## Resource checksums Charmed OpenSearch and Charmed OpenSearch Dashboards operators use pinned revisions of their respective snaps to provide reproducible and secure environments. The [Charmed OpenSearch snap](https://snapstore.io/opensearch) and [Charmed OpenSearch Dashboards snap](https://snapstore.io/opensearch-dashboards) package the OpenSearch and OpenSearch Dashboards workloads, respectively, along with the necessary dependencies and utilities for operator lifecycle management. For details on the contents of these snaps, refer to the `snapcraft.yaml` file in the source code: [Charmed OpenSearch snap contents](https://github.com/canonical/opensearch-snap/blob/2/edge/snap/snapcraft.yaml) and [Charmed OpenSearch Dashboards snap contents](https://github.com/canonical/opensearch-dashboards-snap/blob/2/edge/snap/snapcraft.yaml). Every artifact included in the snaps is verified against its SHA-256 or SHA-512 checksum after download. ## Sources verification Charmed OpenSearch sources are stored in: - GitHub repositories for snaps, rocks and charms - Launchpad repositories for the OpenSearch and OpenSearch Dashboards upstream fork used for building their respective distributions ### Launchpad Distributions are built using private repositories only, hosted as part of the [SOSS namespace](https://launchpad.net/soss) to eventually integrate with Canonical’s standard process for fixing CVEs. Branches associated with releases are mirrored to a public repository, hosted in the [Data Platform namespace](https://launchpad.net/~data-platform) to also provide the community with the patched source code. ### GitHub All OpenSearch artifacts built by Canonical are published and released programmatically using release pipelines implemented via GitHub Actions. Distributions are published as both GitHub and Launchpad releases via the [central-uploader repository](https://github.com/canonical/central-uploader), while charms, snaps and rocks are published using the workflows of their respective repositories. All repositories in GitHub are set up with branch protection rules, requiring: - new commits to be merged to main branches via pull request with at least 2 approvals from repository maintainers - new commits to be signed (e.g. using GPG keys) - developers to sign the [Canonical Contributor License Agreement (CLA)](https://ubuntu.com/legal/contributors) ## Encryption Charmed OpenSearch can be used to deploy a secure OpenSearch cluster that provides encryption-in-transit capabilities out of the box for: - Internode communications - OpenSearch Dashboard connection - External client connection To set up a secure connection Charmed OpenSearch and Charmed OpenSearch Dashboards applications need to be integrated with TLS Certificate Provider charms, e.g. self-signed-certificates operator. Certificate Signing Requests (CSRs) are generated for every unit using the tls_certificates_interface library that uses the cryptography Python library to create X.509 compatible certificates. The CSR is signed by the TLS Certificate Provider, returned to the units, and stored in a password-protected Keystore file. The password of the Keystore is stored in Juju secrets. The relation also provides the CA certificate, which is loaded into a password-protected Truststore file. When encryption is enabled, hostname verification is turned on for client connections, including inter-node communication. The cipher suite can be customized by specifying a list of allowed cipher suites for external clients and OpenSearch Dashboards connections. Encryption at rest is currently not supported, although it can be provided by the substrate (cloud or on-premises). ## Authentication In Charmed OpenSearch, authentication layers can be enabled for: 1. OpenSearch Dashboards connections 2. OpenSearch inter-node communication 3. OpenSearch clients ### OpenSearch authentication to OpenSearch Dashboards Authentication to OpenSearch Dashboards is based on HTTP basic authentication with username and password and implemented both for client-server (with OpenSearch) and server-server communication. Username and passwords are exchanged using peer relations among OpenSearch units and using normal relations between OpenSearch and Opensearch Dashboards. The file needs to be readable and writable by root (as it is created by the charm) and readable by the snap_daemon user running the OpenSearch server snap commands. ### OpenSearch inter-node authentication Authentication among nodes is based on the HTTP basic authentication with username and password. Usernames and passwords are exchanged via peer relations. The OpenSearch username and password, used by nodes to authenticate one another, are stored in an OpenSearch configuration file in plain text format. The file needs to be readable and writable by root (as it is created by the charm) and readable by the snap_daemon user running the OpenSearch server snap commands. ### Client authentication to OpenSearch Authentication among nodes is based on the HTTP basic authentication with username and password. Usernames and passwords are exchanged via peer relations. Usernames and passwords are stored in OpenSearch to be used by the OpenSearch processes and their filesystem structure, in peer-relation data to be used by the OpenSearch charm and in external relation to be shared with client applications.