Confluent kafka operator

Confluent kafka operator DEFAULT

Deploy Confluent Platform¶

After you complete all the required configuration tasks, deploy Confluent Platform components as described in this topic.

Specify the Confluent Platform images and version to deploy¶

  1. For the Confluent Platform components you want to deploy, specify the Docker image names and the version tags in the components custom resources (CRs).

    See Confluent Docker Image Reference for the list of Confluent images.

    The tags are the version of Confluent Platform. For example, for ZooKeeper version 6.2.0, set in the ZooKeeper CR.

  2. Specify the init container in . The tag of the init container is the version of CFK.

    To deploy the 6.2.0 and later versions of Confluent Platform, you must use the instead of the legacy .

    An example ZooKeeper CR to deploy ZooKeeper 6.2.0 with CFK 2.1.0:

    apiVersion:platform.confluent.io/v1beta1kind:Zookeepermetadata:name:zookeepernamespace:confluentspec:image:application:confluentinc/cp-zookeeper:6.2.0init:confluentinc/confluent-init-container:2.1.0

See how to specify different Confluent Platform versions in the component custom resources.

Deploy Confluent Platform¶

  1. Apply the Confluent Platform CR configurations. You can apply one configuration file for all Confluent Platform components, or apply multiple configuration files for individual Confluent Platform component.

    kubectl apply -f <CR>
  2. Check that all Confluent Platform resources are successfully deployed:

© Copyright , Confluent, Inc. Privacy Policy | Terms & Conditions. Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. All other trademarks, servicemarks, and copyrights are the property of their respective owners.

Sours: https://docs.confluent.io/operator/current/co-deploy-cp.html

Confluent for Kubernetes is now Generally Available! You can find the docs at https://docs.confluent.io/operator/current/overview.html and the examples repo at https://github.com/confluentinc/confluent-kubernetes-examples.

[OLD] Confluent Operator Early Access

The next version of Confluent Operator offers a Kubernetes-native experience, including:

  • First class Kubernetes custom resource definitions (CRDs)
  • Config overrides for all Confluent components
  • Auto generated certificates
  • Kubernetes tolerations, pod/node affinity support

Check out the product in action: Kubernetes-Native DevOps with Confluent Operator

Get early access by registering interest here: Confluent Operator Early Access Registration

This repository is structured into Concepts and Scenario workflows:

  • Read Concepts to understand how Confluent on Kubernetes works, and should be architected.
  • Use Scenario workflows to deploy and manage Confluent on Kubernetes for various use cases.

Scenario workflows

Clone the following repo to get the files needed for each workflow:

git clone [email protected]:confluentinc/operator-earlyaccess.git

The documentation is organized as scenario workflows:

Pre-requisites

This Confluent Operator Early Access is compatible with Confluent Platform 6.0.

To use this Confluent Operator Early Access, you’ll need:

  • A Kubernetes cluster - any CNCF conformant version
  • Helm 3 installed on your local machine
  • Kubectl installed on your local machine

Set up the Kubernetes cluster

Set up the Kubernetes cluster for this tutorial.

  1. Add or get access to a Kubernetes cluster.

  2. Create the namespace and set it to the current namespace. In this tutorial, we will deploy Confluent Platform in the namespace.

    kubectl create namespace confluent kubectl config set-context --current --namespace=confluent

Configure Early Access credentials

  1. For this Early Access program, you will have received an API key (associated with your email address) to the Confluent JFrog Artifactory.

  2. Save the API key and your email for easy access while running this tutorial:

    export USER=<your email address> export EMAIL=<your email address> export APIKEY=<your API key>
  3. Create a Kubernetes secret with the registry credentials:

    kubectl create secret docker-registry confluent-registry \ --docker-server=confluent-docker-internal-early-access-operator-2.jfrog.io \ --docker-username=$USER \ --docker-password=$APIKEY \ --docker-email=$EMAIL
  4. The Confluent Operator itself is packaged as a Helm Chart. Add a Helm repo:

    helm repo add confluentinc_earlyaccess \ https://confluent.jfrog.io/confluent/helm-early-access-operator-2 \ --username $USER \ --password $APIKEY helm repo update

Download Confluent Operator tutorial package

Download the tutorial package from the Git Hub repo:

git clone [email protected]:confluentinc/operator-earlyaccess.git

Remove Confluent Operator tutorial environment

When you finish running the scenarios in the Confluent Operator tutorial packages, remove the namespace, the secret, and the Helm repo with the following command:

kubectl delete secret docker-registry helm repo remove confluentinc_earlyaccess kubectl delete namespace confluent
Sours: https://github.com/confluentinc/operator-earlyaccess
  1. Intermotor parts
  2. Tensorflow train model
  3. Sql plus tutorial
  4. Iu patch

Confluent for Kubernetes Quickstart¶

Prerequisites¶

  • A Kubernetes cluster conforming to one of the supported versions.
  • kubectl installed, initialized, with the context set. You also must have the file configured for your cluster.
  • Helm 3 installed.
  • Access to the Confluent for Kubernetes bundle.
  • For this quick start guide, your Kubernetes cluster is assumed to have a default dynamic storage provisioner.

Step 1: Create a namespace¶

  1. Create the namespace to use.

    kubectl create namespace confluent
  2. Set this namespace to default for your Kubernetes context.

    kubectl config set-context --current --namespace confluent

Step 2: Install Confluent for Kubernetes¶

  1. Add the Confluent for Kubernetes Helm repository.

    helm repo add confluentinc https://packages.confluent.io/helm helm repo update
  2. Install Confluent for Kubernetes.

    helm upgrade --install confluent-operator confluentinc/confluent-for-kubernetes

Step 3: Install Confluent Platform¶

  1. Install all Confluent Platform components.

    kubectl apply -f https://raw.githubusercontent.com/confluentinc/confluent-kubernetes-examples/master/quickstart-deploy/confluent-platform.yaml
  2. Install a sample producer app and topic.

    kubectl apply -f https://raw.githubusercontent.com/confluentinc/confluent-kubernetes-examples/master/quickstart-deploy/producer-app-data.yaml
  3. Check that everything is deployed:

Step 4: View Control Center¶

Use Control Center to monitor the Confluent Platform, and see the created topic and data.

  1. Set up port forwarding to Control Center web UI from local machine:

    kubectl port-forward controlcenter-0 9021:9021
  2. Browse to Control Center:

    http://localhost:9021

  3. Check that the topic was created and that messages are being produced to the topic.

Bonus: Install and use Confluent kubectl plugin¶

The Confluent for Kubernetes (CFK) bundle contains a Confluent kubectl plugin for interacting with Confluent for Kubernetes.

  1. Download and unpack the CFK bundle from the following location:

    https://confluent-for-kubernetes.s3-us-west-1.amazonaws.com/confluent-for-kubernetes-2.1.0.tar.gz

  2. Unpack the kubectl plugin that matches your client environment, Linux, Windows or Mac OS (Darwin), into your client environment local executables directory. On Mac OS and Linux, this would be . This will allow the standard CLI to find the plugin.

    tar -xvf kubectl-plugin/kubectl-confluent-<environment>-amd64.tar.gz \ -C <local directory for the plugin>

    For example, on Mac OS:

    tar -xvf kubectl-plugin/kubectl-confluent-darwin-amd64.tar.gz \ -C /usr/local/bin/
  3. Open Confluent Control Center.

    kubectl confluent dashboard controlcenter
  4. View the Confluent Platform version

    kubectl confluent version

© Copyright , Confluent, Inc. Privacy Policy | Terms & Conditions. Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. All other trademarks, servicemarks, and copyrights are the property of their respective owners.

Sours: https://docs.confluent.io/operator/current/co-quickstart.html
Installation - Confluent Operator (2/3)

Confluent Operator¶

Confluent Operator allows you to deploy and manage Confluent Platform as a cloud-native, stateful container application on Kubernetes and OpenShift. The automation provided by Kubernetes, Operator, and Helm greatly simplifies provisioning and minimizes the burden of operating and managing Confluent Platform clusters. Operator also provides you with the portability to use Apache Kafka® in multiple provider zones and across both your private and public cloud environments.

The following video provides an introduction to Confluent Operator. See Introduction to Kubernetes and Confluent Operator for additional Kubernetes and Confluent Operator videos.

Features¶

Deploying
  • Automates Confluent Platform deployment using Helm.
  • Automates Confluent Platform management.
  • Uses default Kubernetes StorageClass-based provisioners for persistent volume storage (such as Google persistent disks, AWS EBS, and Azure disk storage).
  • Automates cluster authentication and security configuration using:
    • SASL/PLAIN with TLS certificates
    • TLS two-way (mutual) authentication
    • TLS in-transit encryption for all deployed components.
Updating and upgrading
  • Provides automated rolling updates for configuration changes.
  • Provides automated rolling upgrades with no impact to Kafka availability.
Scaling
Provides elastic scaling for new Confluent Platform components (with manual rebalancing).
Resiliency
Restores a Kafka pod with the same Kafka broker ID, configuration, and persistent storage volumes if a failure occurs.
Monitoring
  • Supports metrics aggregation using JMX/Jolokia.
  • Supports aggregated metrics export to Prometheus.

Supported environments¶

Operator 1.5.1 is released with Confluent Platform 5.5. For the compatibility information with any other versions of Confluent Platform, see Kubernetes.

Confluent Operator automates the deployment and management of Confluent Platform in the following environments:

  • Amazon Elastic Kubernetes Service (EKS) cluster running on Amazon Web Services (AWS)
  • Azure Kubernetes Service (AKS) cluster running on Microsoft Azure (Azure)
  • Google Kubernetes Engine (GKE) cluster running on Google Cloud Platform (GCP)
  • Red Hat OpenShift Container Platform (3.13 or later) (see note below)
  • Pivotal Container Service (PKS) (1.2 to 1.6)
  • Vanilla Kubernetes (1.13 to 1.15) cluster

Important

Red Hat OpenShift support: Red Hat includes official support for Helm 3 starting with OpenShift version 4.4. If you are using an older version of OpenShift, you need to use Helm 2 to deploy Operator and Confluent Platform 5.5. After downloading the bundle, click the v5.3.2 tab at the bottom of this page and use the Helm 2 instructions for OpenShift deployments.

Architecture¶

The following shows the high-level architecture of Operator and Confluent Platform components in a Kubernetes cluster.

../../_images/co-architecture.png

Hands-on demos¶

The following demos show you deployments of Confluent Platform on Google Kubernetes Engine (GKE) leveraging Confluent Operator:

Licensing¶

You can use Confluent Operator and Confluent Control Center for a 30-day trial period without a license key.

After 30 days, Operator and Control Center require a license key. Confluent issues keys to subscribers, along with providing enterprise-level support for Confluent Platform and Confluent Operator. If you are a subscriber, please contact Confluent Support at support@confluent.io for more information.

See Add a license key if you have received a key for Confluent Operator.

See also

To get started with Confluent Operator on Kubernetes, try out the Kubernetes Demos.

© Copyright , Confluent, Inc. Privacy Policy | Terms & Conditions. Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. All other trademarks, servicemarks, and copyrights are the property of their respective owners.


Last updated on Oct 12, 2021.

Built with Sphinx using a theme provided by Read the Docs.
Sours: https://docs.confluent.io/5.5.1/installation/operator/index.html

Kafka operator confluent

Confluent Operator¶

Confluent Operator allows you to deploy and manage Confluent Platform as a cloud-native, stateful container application on Kubernetes and OpenShift. The automation provided by Kubernetes, Operator, and Helm greatly simplifies provisioning and minimizes the burden of operating and managing Confluent Platform clusters. Operator also provides you with the portability to use Apache Kafka® in multiple provider zones and across both your private and public cloud environments.

Tip

Take a sneak peak into the next generation of Operator.

The following video provides an introduction to Confluent Operator. See Introduction to Kubernetes and Confluent Operator for additional Kubernetes and Confluent Operator videos.

Features¶

Deploying
  • Automates Confluent Platform deployment using Helm.
  • Automates Confluent Platform management.
  • Uses default Kubernetes StorageClass-based provisioners for persistent volume storage (such as Google persistent disks, AWS EBS, and Azure disk storage).
  • Automates cluster authentication and security configuration using:
    • SASL/PLAIN with TLS certificates
    • TLS two-way (mutual) authentication
    • TLS in-transit encryption for all deployed components.
Updating and upgrading
  • Provides automated rolling updates for configuration changes.
  • Provides automated rolling upgrades with no impact to Kafka availability.
Scaling
Provides elastic scaling for new Confluent Platform components.
Resiliency
Restores a Kafka pod with the same Kafka broker ID, configuration, and persistent storage volumes if a failure occurs.
Monitoring
  • Supports metrics aggregation using JMX/Jolokia.
  • Supports aggregated metrics export to Prometheus.

Supported environments¶

Kubernetes versions¶

Confluent Operator 1.7.0 supports Kubernetes versions 1.16 to 1.19 with any Cloud Native Computing Foundation (CNCF) conformant offering. A full list of CNCF conformant Kubernetes offerings can be found here.

The following API must be present in your Kubernetes:

apiextensions.k8s.io/v1beta1

You can run the following command to check. The output should include :

kubectl api-versions | grep apiextensions.k8s.io

Important

Red Hat OpenShift support: Red Hat includes official support for Helm 3 starting with OpenShift version 4.4. If you are using an older version of OpenShift, you need to use Helm 2 to deploy Confluent Platform using Operator 1.7. After downloading the bundle, select cp-5.3.2 from the version picker on this page and use the Helm 2 instructions for OpenShift deployments.

Confluent Platform versions¶

Confluent Operator 1.7.0 supports Confluent Platform versions 6.0.x and 6.1.x.

Architecture¶

The following shows the high-level architecture of Operator and Confluent Platform components in a Kubernetes cluster.

_images/co-architecture.png

Licensing¶

You can use Confluent Operator and Confluent Control Center for a 30-day trial period without a license key.

After 30 days, Operator and Control Center require a license key. Confluent issues keys to subscribers, along with providing enterprise-level support for Confluent Platform and Confluent Operator. If you are a subscriber, please contact Confluent Support at support@confluent.io for more information.

See Add a license key if you have received a key for Confluent Operator.

© Copyright , Confluent, Inc. Privacy Policy | Terms & Conditions. Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. All other trademarks, servicemarks, and copyrights are the property of their respective owners.

Sours: https://docs.confluent.io/operator/1.7.0/overview.html
Helm 101: Apache Kafka and Confluent Platform for Kubernetes Explained

Confluent for Kubernetes¶

Confluent for Kubernetes (CFK) is a cloud-native control plane for deploying and managing Confluent in your private cloud environment. It provides standard and simple interface to customize, deploy, and manage Confluent Platform through declarative API.

Confluent for Kubernetes runs on Kubernetes, the runtime for private cloud architectures.

Note

Confluent for Kubernetes is the next generation of Confluent Operator. For Confluent Operator 1.x documentation, see Confluent Operator 1, or use the version picker to browse to a specific version of the documentation.

See Introducing Confluent for Kubernetes for an overview.

The following shows the high-level architecture of Confluent for Kubernetes and Confluent Platform in Kubernetes.

_images/co-architecture.png

Features¶

The following are summaries of the main, notable features of Confluent for Kubernetes.

Cloud Native Declarative API
  • Declarative Kubernetes-native API approach to configure, deploy, and manage Confluent Platform components (Apache Kafka®, Connect workers, ksqlDB, Schema Registry, Confluent Control Center) and resources (topics, rolebindings) through Infrastructure as Code (IaC).
  • Provides built-in automation for cloud-native security best practices:
    • Complete granular RBAC, authentication and TLS network encryption
    • Auto-generated certificates
    • Support for credential management systems, such as Hashicorp Vault, to inject sensitive configurations in memory to Confluent deployments
  • Provides server properties, JVM, and Log4j configuration overrides for customization of all Confluent Platform components.
Upgrades
  • Provides automated rolling updates for configuration changes.
  • Provides automated rolling upgrades with no impact to Kafka availability.
Scaling
  • Provides single command, automated scaling and reliability checks of Confluent Platform.
Resiliency
  • Restores a Kafka pod with the same Kafka broker ID, configuration, and persistent storage volumes if a failure occurs.
  • Provides automated rack awareness to spread replicas of a partition across different racks (or zones), improving availability of Kafka brokers and limiting the risk of data loss.
Scheduling
  • Supports Kubernetes labels and annotations to provide useful context to DevOps teams and ecosystem tooling.
  • Supports Kubernetes tolerations and pod/node affinity for efficient resource utilization and pod placement.
Monitoring
  • Supports metrics aggregation using JMX/Jolokia.
  • Supports aggregated metrics export to Prometheus.

Licensing¶

You can use Confluent for Kubernetes and Confluent Control Center for a 30-day trial period without a license key.

After 30 days, Confluent for Kubernetes and Control Center require a license key. Confluent issues keys to subscribers, along with providing enterprise-level support for Confluent components and Confluent for Kubernetes.

If you are a subscriber, contact Confluent Support at support@confluent.io for more information.

See Update Confluent Platform License if you have received a key for Confluent for Kubernetes.

© Copyright , Confluent, Inc. Privacy Policy | Terms & Conditions. Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. All other trademarks, servicemarks, and copyrights are the property of their respective owners.

Sours: https://docs.confluent.io/operator/current/overview.html

You will also be interested:

Is groaning somewhere under my left buttock, as well as what and when I was ill. - How often do you live sexually. - asked the doctor. Here they went, tricky questions.



611 612 613 614 615