Introducción
Características principales de Redpanda
Único binario
Compatibilidad con Kafka
Consola de administración
Gran rendimiento
Facil administración
Despliegue en cualquier entorno
Puesta en marcha
version: "3.7" name: redpanda-quickstart-one-broker networks: redpanda_network: driver: bridge volumes: redpanda-0: null services: redpanda-0: command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092 # Address the broker advertises to clients that connect to the Kafka API. # Use the internal addresses to connect to the Redpanda brokers' # from inside the same Docker network. # Use the external addresses to connect to the Redpanda brokers' # from outside the Docker network. - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092 - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082 # Address the broker advertises to clients that connect to the HTTP Proxy. - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082 - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081 # Redpanda brokers use the RPC API to communicate with each other internally. - --rpc-addr redpanda-0:33145 - --advertise-rpc-addr redpanda-0:33145 # Mode dev-container uses well-known configuration properties for development in containers. - --mode dev-container # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system. - --smp 1 - --default-log-level=info image: docker.redpanda.com/redpandadata/redpanda:v23.3.9 container_name: redpanda-0 volumes: - redpanda-0:/var/lib/redpanda/data networks: - redpanda_network ports: - 18081:18081 - 18082:18082 - 19092:19092 - 19644:9644 console: container_name: redpanda-console image: docker.redpanda.com/redpandadata/console:v2.4.5 networks: - redpanda_network entrypoint: /bin/sh command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console' environment: CONFIG_FILEPATH: /tmp/config.yml CONSOLE_CONFIG_FILE: | kafka: brokers: ["redpanda-0:9092"] schemaRegistry: enabled: true urls: ["http://redpanda-0:8081"] redpanda: adminApi: enabled: true urls: ["http://redpanda-0:9644"] ports: - 8081:8080 depends_on: - redpanda-0
docker compose up -d
[+] Running 2/2 ⠿ Container redpanda-0 Started 0.2s ⠿ Container redpanda-console Started 0.3s
Obtener información sobre el cluster de Redpanda:
Una vez arrancado el servicio con docker-compose, podemos ver el estado del cluster con el siguiente comando.
docker exec -it redpanda-0 rpk cluster info
A continuación vemos la salida del comando ejecutado donde aparece información básica del cluster que está corriendo, información de los brokers e información de los topics (más tarde veremos como podemos consultar esta información a través de la consola de administración:
CLUSTER ======= redpanda.c3cf5e77-ebbc-40bd-8460-e48e59932523 BROKERS ======= ID HOST PORT 0* redpanda-0 9092 TOPICS ====== NAME PARTITIONS REPLICAS _schemas 1 1
Crear un topic llamado «test-topic»
Para crear un topic llamado test-topic, bastará con lanzar el siguiente comando:
docker exec -it redpanda-0 rpk topic create test-topic
Salida del comando:
TOPIC STATUS test-topic OK
Producir un mensaje
docker exec -it redpanda-0 rpk topic produce test-topic
Esto es un mensaje de prueba Produced to partition 0 at offset 0 with timestamp 1711011295492.
Consumir un mensaje
docker exec -it redpanda-0 rpk topic consume test-topic --num 1
{ "topic": "test-topic", "value": "Esto es un mensaje de prueba", "timestamp": 1711011295492, "partition": 0, "offset": 0 }
Los ejemplos que hemos visto anteriormente son muy sencillos y, por supuesto, hay muchos más comandos con los cuales administrar Redpanda. En el siguiente enlace te dejo la referencia completa de los comandos de rpk.
Consola de administración
Vamos a lo que nos interesa, integrémoslo con una aplicación
spring.application.name=redpanda ## KAFKA spring.kafka.bootstrap-servers=localhost:19092 ## KAFKA PRODUCER spring.kafka.producer.bootstrap-servers=localhost:19092 spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer spring.kafka.producer.value-serializer=com.example.redpanda.library.domainevents.KafkaEventSerializer ## KAFKA CONSUMER spring.kafka.consumer.bootstrap-servers=localhost:19092 spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer spring.kafka.consumer.value-deserializer=com.example.redpanda.library.domainevents.KafkaEventDeserializer spring.kafka.consumer.auto-offset-reset=earliest spring.kafka.consumer.enable-auto-commit=true spring.kafka.consumer.group-id=redpanda-app
Hay un broker de Kafka escuchando en el puerto 19092 y tenemos dado de alta el producer y el consumer con un serializer y deserializer customs.
Pues bien, la prueba ha consistido en tirar el broker de Kafka y levantar el broker de Redpanda escuchando en el mismo puerto para comprobar que efectivamente Redpanda se integra perfectamente con nuestra aplicación y nos proporciona una correcta gestión de los eventos de nuestra aplicación.
2024-03-22T11:26:43.871+01:00 INFO 84868 --- [redpanda] [ main] o.a.kafka.common.utils.AppInfoParser : Kafka version: 3.6.1 2024-03-22T11:26:43.871+01:00 INFO 84868 --- [redpanda] [ main] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: 5e3c2b738d253ff5 2024-03-22T11:26:43.871+01:00 INFO 84868 --- [redpanda] [ main] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1711103203871 2024-03-22T11:26:43.871+01:00 INFO 84868 --- [redpanda] [ main] fkaConsumerFactory$ExtendedKafkaConsumer : [Consumer clientId=consumer-redpanda-app-3, groupId=redpanda-app] Subscribed to topic(s): users.user.deleted 2024-03-22T11:26:43.872+01:00 INFO 84868 --- [redpanda] [ntainer#2-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-redpanda-app-1, groupId=redpanda-app] Request joining group due to: need to re-join with the given member-id: consumer-redpanda-app-1-ac6b4094-3264-4e2a-be8e-13cd739fdb11 2024-03-22T11:26:43.872+01:00 INFO 84868 --- [redpanda] [ntainer#2-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-redpanda-app-1, groupId=redpanda-app] Request joining group due to: rebalance failed due to 'The group member needs to have a valid member id before actually entering a consumer group.' (MemberIdRequiredException) 2024-03-22T11:26:43.872+01:00 INFO 84868 --- [redpanda] [ntainer#2-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-redpanda-app-1, groupId=redpanda-app] (Re-)joining group 2024-03-22T11:26:43.872+01:00 INFO 84868 --- [redpanda] [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-redpanda-app-2, groupId=redpanda-app] Request joining group due to: need to re-join with the given member-id: consumer-redpanda-app-2-1096e52d-79e9-4c37-8ea0-621a53b3ff4a 2024-03-22T11:26:43.872+01:00 INFO 84868 --- [redpanda] [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-redpanda-app-2, groupId=redpanda-app] Request joining group due to: rebalance failed due to 'The group member needs to have a valid member id before actually entering a consumer group.' (MemberIdRequiredException) 2024-03-22T11:26:43.872+01:00 INFO 84868 --- [redpanda] [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-redpanda-app-2, groupId=redpanda-app] (Re-)joining group 2024-03-22T11:26:43.874+01:00 INFO 84868 --- [redpanda] [ntainer#1-0-C-1] org.apache.kafka.clients.Metadata : [Consumer clientId=consumer-redpanda-app-3, groupId=redpanda-app] Cluster ID: redpanda.c3cf5e77-ebbc-40bd-8460-e48e59932523 2024-03-22T11:26:43.874+01:00 INFO 84868 --- [redpanda] [ntainer#1-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-redpanda-app-3, groupId=redpanda-app] Discovered group coordinator localhost:19092 (id: 2147483647 rack: null) 2024-03-22T11:26:43.875+01:00 INFO 84868 --- [redpanda] [ntainer#1-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-redpanda-app-3, groupId=redpanda-app] (Re-)joining group 2024-03-22T11:26:43.876+01:00 INFO 84868 --- [redpanda] [ntainer#1-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-redpanda-app-3, groupId=redpanda-app] Request joining group due to: need to re-join with the given member-id: consumer-redpanda-app-3-b2b59b71-3b62-4c5c-b9ab-df39c067f6c0 2024-03-22T11:26:43.876+01:00 INFO 84868 --- [redpanda] [ntainer#1-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-redpanda-app-3, groupId=redpanda-app] Request joining group due to: rebalance failed due to 'The group member needs to have a valid member id before actually entering a consumer group.' (MemberIdRequiredException) 2024-03-22T11:26:43.876+01:00 INFO 84868 --- [redpanda] [ntainer#1-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-redpanda-app-3, groupId=redpanda-app] (Re-)joining group 2024-03-22T11:26:43.878+01:00 INFO 84868 --- [redpanda] [ main] c.example.redpanda.RedpandaApplication : Started RedpandaApplication in 2.086 seconds (process running for 2.36)
2024-03-22T11:29:06.760+01:00 INFO 84868 --- [redpanda] [nio-8080-exec-1] o.a.k.clients.producer.KafkaProducer : [Producer clientId=producer-1] Instantiated an idempotent producer. 2024-03-22T11:29:06.765+01:00 INFO 84868 --- [redpanda] [nio-8080-exec-1] o.a.kafka.common.utils.AppInfoParser : Kafka version: 3.6.1 2024-03-22T11:29:06.765+01:00 INFO 84868 --- [redpanda] [nio-8080-exec-1] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: 5e3c2b738d253ff5 2024-03-22T11:29:06.765+01:00 INFO 84868 --- [redpanda] [nio-8080-exec-1] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1711103346765 2024-03-22T11:29:06.769+01:00 INFO 84868 --- [redpanda] [ad | producer-1] org.apache.kafka.clients.Metadata : [Producer clientId=producer-1] Cluster ID: redpanda.c3cf5e77-ebbc-40bd-8460-e48e59932523 2024-03-22T11:29:06.774+01:00 INFO 84868 --- [redpanda] [ad | producer-1] o.a.k.c.p.internals.TransactionManager : [Producer clientId=producer-1] ProducerId set to 4001 with epoch 0 2024-03-22T11:29:06.807+01:00 INFO 84868 --- [redpanda] [ntainer#0-0-C-1] c.e.r.c.u.p.e.UserCreatedEventSubscriber : Message received: UserCreatedEvent{id='70f2d981-61d6-491a-9093-7bcc74e02afd', name='Ismael'} 2024-03-22T11:29:14.584+01:00 INFO 84868 --- [redpanda] [ntainer#2-0-C-1] c.e.r.c.u.p.e.UserUpdatedEventSubscriber : Message received: UserUpdatedEvent{id='70f2d981-61d6-491a-9093-7bcc74e02afd', name='Ismael Fernandez'} 2024-03-22T11:29:20.389+01:00 INFO 84868 --- [redpanda] [ntainer#1-0-C-1] c.e.r.c.u.p.e.UserDeletedEventSubscriber : Message received: UserDeletedEvent{id='70f2d981-61d6-491a-9093-7bcc74e02afd', name='Ismael Fernandez'}
Conclusiones