PUBLICKEY: <...>kind: Secretmetadata: creationTimestamp: null name: pulsar-token-asymmetric-key namespace: pulsargenerate the tokens for the super-users: proxy-admin,broker-admin,admingenerate the token for proxy-admin---pulsar-token-asymmetric-keyapiVersion: v1data: TOKEN: <...> TYPE: YXN5bW1ldHJpYw==kind: Secretmetadata: creationTimestamp: null name: pulsar-token-proxy-admin namespace: pulsargenerate the token for broker-admin---pulsar-token-asymmetric-keyapiVersion: v1data: TOKEN: <...> TYPE: YXN5bW1ldHJpYw==kind: Secretmetadata: creationTimestamp: null name: pulsar-token-broker-admin namespace: pulsargenerate the token for admin---pulsar-token-asymmetric-keyapiVersion: v1data: TOKEN: <...> TYPE: YXN5bW1ldHJpYw==kind: Secretmetadata: creationTimestamp: null name: pulsar-token-admin namespace: pulsar-------------------------------------The jwt token secret keys are generated under: - 'pulsar-token-asymmetric-key'The jwt tokens for superusers are generated and stored as below: - 'proxy-admin':secret('pulsar-token-proxy-admin') - 'broker-admin':secret('pulsar-token-broker-admin') - 'admin':secret('pulsar-token-admin')从输出可以看出，该脚本生成了4个K8S Secret的Manifest:pulsar-token-asymmetric-key这个Secret中是用于生成Token和验证Token的私钥和公钥pulsar-token-proxy-admin这个Secret中是用于proxy的超级用户角色Tokenpulsar-token-broker-admin这个Secret中是用于broker的超级用户角色Tokenpulsar-token-admin这个Secret中是用于管理客户端的超级用户角色Token接下来手动将这4个Secret使用kubectl apply创建到K8S的pulsar命名空间中。创建完成后，可以使用kubectl找到它们:kubectl get secret -n pulsar | grep pulsar-tokenpulsar-token-admin Opaque 2 5mpulsar-token-asymmetric-key Opaque 2 5mpulsar-token-broker-admin Opaque 2 5mpulsar-token-proxy-admin Opaque 2 5m1.4 创建Zookeeper和Bookie的Local PV根据部署Pulsar的K8S节点的规划，下面需要为zookeeper, bookie所在的节点在K8S上创建Local Persistent Volume。注意每个zookeeper节点需要一个data的local volume，每个bookie节点需要journal和ledgers共两个local volume。在创建Local PV之前，需要确认一下k8s中存在StorageClasslocal-storage，如果没有可以使用下面的manifest创建。apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: local-storageprovisioner: kubernetes.io/no-provisionervolumeBindingMode: WaitForFirstConsumerreclaimPolicy: Retain注意现在的K8S中不在直接提供local volume的provisioner，这里也没有使用provisioner，因此后续对local volume的创建和管理都是需要K8S集群管理员的手动进行。也是说目前Kubernetes核心中不包含对对本地卷进行动态发放和管理的provisioner，如果想要体验动态发放和管理的功能，可以试一下由Rancher提供的Local Path Provisioner。我这里依然使用手动管理的方式，即通过手动在K8S节点上创建Local Volume，手动绑定Local Volume与Pulsar Zookeeper和Bookie的PVC(PersistentVolumeClaim)之间的关系。下面，先手动在node1, node2, node3上创建local volume对应的数据目录:mkdir -p /home/puslar/data/zookeeper-datamkdir -p /home/puslar/data/bookie-data/ledgersmkdir -p /home/puslar/data/bookie-data/journalzookeeper data的local pv的manifest如下:---apiVersion: v1kind: PersistentVolumemetadata: name: pulsar-zookeeper-data-pulsar-zookeeper-0spec: capacity: storage: 20Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/puslar/data/zookeeper-data claimRef: name: pulsar-zookeeper-data-pulsar-zookeeper-0 namespace: pulsar nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node1---apiVersion: v1kind: PersistentVolumemetadata: name: pulsar-zookeeper-data-pulsar-zookeeper-1spec: capacity: storage: 20Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/puslar/data/zookeeper-data claimRef: name: pulsar-zookeeper-data-pulsar-zookeeper-1 namespace: pulsar nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node2---apiVersion: v1kind: PersistentVolumemetadata: name: pulsar-zookeeper-data-pulsar-zookeeper-2spec: capacity: storage: 20Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/puslar/data/zookeeper-data claimRef: name: pulsar-zookeeper-data-pulsar-zookeeper-2 namespace: pulsar nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node3上面的manifest仍中将3个Local PV通过nodeAffinity创建并关联到到node1~node3上，同时使用claimRef将这3个Local PV与即将在K8S集群中部署的zookeeper SatefulSet中的PVC绑定。使用kubectl apply创建上面的manifest。bookie ledgers和journal的local pv的manifest如下:---apiVersion: v1kind: PersistentVolumemetadata: name: pulsar-bookie-ledgers-pulsar-bookie-0spec: capacity: storage: 50Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/puslar/data/bookie-data/ledgers claimRef: name: pulsar-bookie-ledgers-pulsar-bookie-0 namespace: pulsar nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node1---apiVersion: v1kind: PersistentVolumemetadata: name: pulsar-bookie-journal-pulsar-bookie-0spec: capacity: storage: 50Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/puslar/data/bookie-data/journal claimRef: name: pulsar-bookie-journal-pulsar-bookie-0 namespace: pulsar nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node1---apiVersion: v1kind: PersistentVolumemetadata: name: pulsar-bookie-ledgers-pulsar-bookie-1spec: capacity: storage: 50Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/puslar/data/bookie-data/ledgers claimRef: name: pulsar-bookie-ledgers-pulsar-bookie-1 namespace: pulsar nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node2---apiVersion: v1kind: PersistentVolumemetadata: name: pulsar-bookie-journal-pulsar-bookie-1spec: capacity: storage: 50Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/puslar/data/bookie-data/journal claimRef: name: pulsar-bookie-journal-pulsar-bookie-1 namespace: pulsar nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node2---apiVersion: v1kind: PersistentVolumemetadata: name: pulsar-bookie-ledgers-pulsar-bookie-2spec: capacity: storage: 50Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/puslar/data/bookie-data/ledgers claimRef: name: pulsar-bookie-ledgers-pulsar-bookie-2 namespace: pulsar nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node3---apiVersion: v1kind: PersistentVolumemetadata: name: pulsar-bookie-journal-pulsar-bookie-2spec: capacity: storage: 50Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/puslar/data/bookie-data/journal claimRef: name: pulsar-bookie-journal-pulsar-bookie-2 namespace: pulsar nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node3上面的manifest仍中将6个Local PV通过nodeAffinity创建并关联到到node1~node3上，同时使用claimRef将这3个Local PV与即将在K8S集群中部署的zookeeper SatefulSet中的PVC绑定。使用kubectl apply创建上面的manifest。1.5 准备Pulsar Manager的PostgreSQL数据库这里准备让Pulsar Manager使用外部数据库，需要提前在外部的PostgreSQL中创建好用户和数据库表结构。创建数据库和用户:CREATE USER pulsar_manager WITH PASSWORD '';CREATE DATABASE pulsar_manager OWNER pulsar_manager;GRANT ALL PRIVILEGES ON DATABASE pulsar_manager to pulsar_manager;GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA pulsar_manager TO pulsar_manager;ALTER SCHEMA public OWNER to pulsar_manager;GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO pulsar_manager;GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO pulsar_manager;创建表结构(建表脚本可以在pulsar-manager的镜像中找到):CREATE TABLE IF NOT EXISTS environments ( name varchar(256) NOT NULL, broker varchar(1024) NOT NULL, CONSTRAINT PK_name PRIMARY KEY (name), UNIQUE (broker));CREATE TABLE IF NOT EXISTS topics_stats ( topic_stats_id BIGSERIAL PRIMARY KEY, environment varchar(255) NOT NULL, cluster varchar(255) NOT NULL, broker varchar(255) NOT NULL, tenant varchar(255) NOT NULL, namespace varchar(255) NOT NULL, bundle varchar(255) NOT NULL, persistent varchar(36) NOT NULL, topic varchar(255) NOT NULL, producer_count BIGINT, subscription_count BIGINT, msg_rate_in double precision , msg_throughput_in double precision , msg_rate_out double precision , msg_throughput_out double precision , average_msg_size double precision , storage_size double precision , time_stamp BIGINT);CREATE TABLE IF NOT EXISTS publishers_stats ( publisher_stats_id BIGSERIAL PRIMARY KEY, producer_id BIGINT, topic_stats_id BIGINT NOT NULL, producer_name varchar(255) NOT NULL, msg_rate_in double precision , msg_throughput_in double precision , average_msg_size double precision , address varchar(255), connected_since varchar(128), client_version varchar(36), metadata text, time_stamp BIGINT, CONSTRAINT fk_publishers_stats_topic_stats_id FOREIGN KEY (topic_stats_id) References topics_stats(topic_stats_id));CREATE TABLE IF NOT EXISTS replications_stats ( replication_stats_id BIGSERIAL PRIMARY KEY, topic_stats_id BIGINT NOT NULL, cluster varchar(255) NOT NULL, connected BOOLEAN, msg_rate_in double precision , msg_rate_out double precision , msg_rate_expired double precision , msg_throughput_in double precision , msg_throughput_out double precision , msg_rate_redeliver double precision , replication_backlog BIGINT, replication_delay_in_seconds BIGINT, inbound_connection varchar(255), inbound_connected_since varchar(255), outbound_connection varchar(255), outbound_connected_since varchar(255), time_stamp BIGINT, CONSTRAINT FK_replications_stats_topic_stats_id FOREIGN KEY (topic_stats_id) References topics_stats(topic_stats_id));CREATE TABLE IF NOT EXISTS subscriptions_stats ( subscription_stats_id BIGSERIAL PRIMARY KEY, topic_stats_id BIGINT NOT NULL, subscription varchar(255) NULL, msg_backlog BIGINT, msg_rate_expired double precision , msg_rate_out double precision , msg_throughput_out double precision , msg_rate_redeliver double precision , number_of_entries_since_first_not_acked_message BIGINT, total_non_contiguous_deleted_messages_range BIGINT, subscription_type varchar(16), blocked_subscription_on_unacked_msgs BOOLEAN, time_stamp BIGINT, UNIQUE (topic_stats_id, subscription), CONSTRAINT FK_subscriptions_stats_topic_stats_id FOREIGN KEY (topic_stats_id) References topics_stats(topic_stats_id));CREATE TABLE IF NOT EXISTS consumers_stats ( consumer_stats_id BIGSERIAL PRIMARY KEY, consumer varchar(255) NOT NULL, topic_stats_id BIGINT NOT NUll, replication_stats_id BIGINT, subscription_stats_id BIGINT, address varchar(255), available_permits BIGINT, connected_since varchar(255), msg_rate_out double precision , msg_throughput_out double precision , msg_rate_redeliver double precision , client_version varchar(36), time_stamp BIGINT, metadata text);CREATE TABLE IF NOT EXISTS tokens ( token_id BIGSERIAL PRIMARY KEY, role varchar(256) NOT NULL, description varchar(128), token varchar(1024) NOT NUll, UNIQUE (role));CREATE TABLE IF NOT EXISTS users ( user_id BIGSERIAL PRIMARY KEY, access_token varchar(256), name varchar(256) NOT NULL, description varchar(128), email varchar(256), phone_number varchar(48), location varchar(256), company varchar(256), expire BIGINT, password varchar(256), UNIQUE (name));CREATE TABLE IF NOT EXISTS roles ( role_id BIGSERIAL PRIMARY KEY, role_name varchar(256) NOT NULL, role_source varchar(256) NOT NULL, description varchar(128), resource_id BIGINT NOT NULL, resource_type varchar(48) NOT NULL, resource_name varchar(48) NOT NULL, resource_verbs varchar(256) NOT NULL, flag INT NOT NULL);CREATE TABLE IF NOT EXISTS tenants ( tenant_id BIGSERIAL PRIMARY KEY, tenant varchar(255) NOT NULL, admin_roles varchar(255), allowed_clusters varchar(255), environment_name varchar(255), UNIQUE(tenant));CREATE TABLE IF NOT EXISTS namespaces ( namespace_id BIGSERIAL PRIMARY KEY, tenant varchar(255) NOT NULL, namespace varchar(255) NOT NULL, UNIQUE(tenant, namespace));CREATE TABLE IF NOT EXISTS role_binding( role_binding_id BIGSERIAL PRIMARY KEY, name varchar(256) NOT NULL, description varchar(256), role_id BIGINT NOT NULL, user_id BIGINT NOT NULL);上面已经做好了部署的准备工作，下面将使用Helm在K8S集群中部署Pulsar集群。2.使用Helm在K8S中部署Pulsar从https://github.com/apache/pulsar-helm-chart/releases下载pulsar helm chart 2.7.7到K8S的控制节点上(kubectl和helm可用)。https://github.com/apache/pulsar-helm-chart/releases/download/pulsar-2.7.7/pulsar-2.7.7.tgz2.1 定制编写helm chart的values.yaml定制编写helm chart的values.yaml文件如下, 定制的内容比较多，具体见下面文件的注释:auth: authentication: enabled: true # 开启jwt认证 provider: "jwt" jwt: usingSecretKey: false # jwt认证使用非对称秘钥对 authorization: enabled: true # 开启授权 superUsers: # broker to broker communication broker: "broker-admin" # proxy to broker communication proxy: "proxy-admin" # pulsar-admin client to broker/proxy communication client: "admin"components: # 启用的组件 autorecovery: true bookkeeper: true broker: true functions: true proxy: true pulsar_manager: true toolset: true zookeeper: truemonitoring: # 关闭监控组件, 后续尝试使用外部Prometheus对pulsar集群进行监控 grafana: false prometheus: false node_exporter: falsevolumes: local_storage: true # 数据卷使用local storageproxy: # proxy的配置(这里是测试环境, 将proxy也调度到node1或node2或node3) nodeSelector: node-role.kubernetes.io/pulsar: pulsar tolerations: - key: "dedicated" operator: "Equal" value: "pulsar" effect: "NoSchedule" configData: PULSAR_PREFIX_authenticateMetricsEndpoint: "false"broker: # broker的配置(这里是测试环境, 将proxy也调度到node1或node2或node3) nodeSelector: node-role.kubernetes.io/pulsar: pulsar tolerations: - key: "dedicated" operator: "Equal" value: "pulsar" effect: "NoSchedule" zookeeper: # broker的配置 replicaCount: 3 tolerations: - key: "dedicated" operator: "Equal" value: "pulsar" effect: "NoSchedule" volumes: data: # 配置使用local pv, 需要与前面手动创建的local pv信息一致 local_storage: true size: 20Gibookkeeper: # bookkeeper的配置 replicaCount: 3 tolerations: - key: "dedicated" operator: "Equal" value: "pulsar" effect: "NoSchedule" volumes: journal: # 配置使用local pv, 需要与前面手动创建的local pv信息一致 local_storage: true size: 50Gi ledgers: # 配置使用local pv, 需要与前面手动创建的local pv信息一致 local_storage: true size: 50Gipulsar_manager: # pulsar_manager的配置(这里是测试环境, 将pulsar_manager也调度到node1或node2或node3) replicaCount: 1 admin: # 文档中描述这里是pulsar manager web界面登录用户密码，但实际上当使用外部PostgreSQL数据库时，这里需要指定PostgreSQL的数据库和密码，不知道是否是pulsar-helm-chart 2.7.7的问题 user: pulsar_manager password: 05aM3Braz_M4RWpn configData: DRIVER_CLASS_NAME: org.postgresql.Driver URL: jdbc:postgresql://:5432/pulsar_manager # 文档中描述这里PostgreSQL数据库的密码，但实际上这里不能指定USERNAME和PASSWORD, 不知道是否是pulsar-helm-chart 2.7.7的问题 # USERNAME: pulsar_manager # PASSWORD: 05aM3Braz_M4RWpn LOG_LEVEL: INFO ## 开启JWT认证后, 这里需要指定pulsar-token-admin这个Secret中的JWT Token JWT_TOKEN: autorecovery: # autorecovery的配置(这里是测试环境, 将autorecovery也调度到node1或node2或node3) replicaCount: 1 nodeSelector: node-role.kubernetes.io/pulsar: pulsar tolerations: - key: "dedicated" operator: "Equal" value: "pulsar" effect: "NoSchedule"toolset: # toolset的配置(这里是测试环境, 将toolset也调度到node1或node2或node3) replicaCount: 1 nodeSelector: node-role.kubernetes.io/pulsar: pulsar tolerations: - key: "dedicated" operator: "Equal" value: "pulsar" effect: "NoSchedule"images: # 对个组件使用私有镜像仓库的配置 imagePullSecrets: - regsecret # 私有镜像仓库的image pull secret, 需要提前在k8s命名空间中创建 autorecovery: repository: harbor.example.com/library/apachepulsar/pulsar-all tag: 2.7.4 bookie: repository: harbor.example.com/library/apachepulsar/pulsar-all tag: 2.7.4 broker: repository: harbor.example.com/library/apachepulsar/pulsar-all tag: 2.7.4 functions: repository: harbor.example.com/library/apachepulsar/pulsar-all tag: 2.7.4 proxy: repository: harbor.example.com/library/apachepulsar/pulsar-all tag: 2.7.4 pulsar_manager: repository: harbor.example.com/library/apachepulsar/pulsar-manager tag: v0.2.0 zookeeper: repository: harbor.example.com/library/apachepulsar/pulsar-all tag: 2.7.4pulsar_metadata: component: pulsar-init image: # the image used for running `pulsar-cluster-initialize` job repository: harbor.example.com/library/apachepulsar/pulsar-all tag: 2.7.4因为当前在pulsar-helm-chart 2.7.7 中好像不支持为pulsar-init设置私有仓库的imagePullSecret，所以下面为pulsar namespace中的default servcieaccount 添加上imagePullSecret。kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regsecret"}]}' -n pulsar2.2 使用helm install安装pulsar定制完value.yaml之后，使用下面的命令向K8S集群部署pulsar。helm install \ --values values.yaml \ --set initialize=true \ --namespace pulsar \ pulsar pulsar-2.7.7.tgz安装完成后使用下面的命令查看一下两个初始化job pulsar-pulsar-init和pulsar-bookie-init的pod状态为Complete:kubectl get pod -n pulsar | grep initpulsar-bookie-init--1-h65bp 0/1 Completed 0 5m14spulsar-pulsar-init--1-t4thq 0/1 Completed 0 5m5s使用下面的命令查看一下pulsar集群各个组件的Pod状态全部都为Running:kubectl get pod -n pulsar -l cluster=pulsar -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESpulsar-bookie-0 1/1 Running 0 14m 10.244.226.91 node1 pulsar-bookie-1 1/1 Running 0 14m 10.244.63.90 node2 pulsar-bookie-2 1/1 Running 0 14m 10.244.46.92 node3 pulsar-broker-0 1/1 Running 0 14m 10.244.226.90 node1 pulsar-broker-1 1/1 Running 0 14m 10.244.63.89 node2 pulsar-broker-2 1/1 Running 0 14m 10.244.46.90 node3 pulsar-proxy-0 1/1 Running 0 14m 10.244.226.93 node1 pulsar-proxy-1 1/1 Running 0 14m 10.244.63.91 node2 pulsar-proxy-2 1/1 Running 0 14m 10.244.46.93 node3 pulsar-pulsar-manager-7b98666cff-5626f 1/1 Running 0 14m 10.244.63.88 node2 pulsar-recovery-0 1/1 Running 0 14m 10.244.46.89 node3 pulsar-toolset-0 1/1 Running 0 14m 10.244.46.91 node3 pulsar-zookeeper-0 1/1 Running 0 14m 10.244.226.92 node1 pulsar-zookeeper-1 1/1 Running 0 14m 10.244.63.92 node2 pulsar-zookeeper-2 1/1 Running 0 13m 10.244.46.94 node3 如果后边调整了values.yaml，需要更新部署时，使用下面的命令:helm upgrade pulsar pulsar-2.7.7.tgz \ --namespace pulsar \ -f values.yaml2.3 在toolset pod中测试创建tenant, namespace和topictoolset pod中包含了各种管理和测试pulsar的命令行工具，例如pulsar-admin, pulsar-client等。下面进入toolset pod中，使用pulsar-admin命令行工具测试一下tenant, namespace和topic的创建，进一步确认pulsar集群工作正常。kubectl exec -it -n pulsar pulsar-toolset-0 -- /bin/bashbin/pulsar-admin tenants create test-tenantbin/pulsar-admin tenants list"public""pulsar""test-tenant"bin/pulsar-admin namespaces create test-tenant/test-nsbin/pulsar-admin namespaces list test-tenant"test-tenant/test-ns"bin/pulsar-admin topics create-partitioned-topic test-tenant/test-ns/test-topic -p 3bin/pulsar-admin topics list-partitioned-topics test-tenant/test-ns"persistent://test-tenant/test-ns/test-topic"2.4 创建pulsar-manager的管理员用户并登录查看下面测试一下pulsar manager是否可以使用。前面使用helm chart部署的pulsar集群，在k8s中创建了下面7个Service。kubectl get svc -l app=pulsar -n pulsarNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEpulsar-bookie ClusterIP None 3181/TCP,8000/TCP 40mpulsar-broker ClusterIP None 8080/TCP,6650/TCP 40mpulsar-proxy LoadBalancer 10.104.105.137 80:31970/TCP,6650:32631/TCP 40mpulsar-pulsar-manager LoadBalancer 10.110.207.9 9527:32764/TCP 40mpulsar-recovery ClusterIP None 8000/TCP 40mpulsar-toolset ClusterIP None 40mpulsar-zookeeper ClusterIP None 8000/TCP,2888/TCP,3888/TCP,2181/TCP 40m从上面命令的输出可以看出，bookie, broker, recovery, toolset, zookeeper这5个Service的类型都是ClusterIP的，并且cluser-ip为None，都是Headless的Service，因为它们只需要在k8s集群内部使用。pulsar-proxy和pulsar-pulsar-manager为LoadBalancer类型，并且都配置了NodePort，提供了从K8S集群外部访问的能力。从集群外部访问pulsar-manager的地址是http://node1:32764，第一次访问pulsar manager之前，需要为其创建一个管理用户:CSRF_TOKEN=$(curl http://node1:32764/pulsar-manager/csrf-token)curl \ -H 'X-XSRF-TOKEN: $CSRF_TOKEN' \ -H 'Cookie: XSRF-TOKEN=$CSRF_TOKEN;' \ -H "Content-Type: application/json" \ -X PUT http://node1:32764/pulsar-manager/users/superuser \ -d '{"name": "admin", "password": "pulsar", "description": "test", "email": "username@test.org"}'上面的命令为pulsar-manager创建用户名为admin, 密码为pulsar的管理用户。使用该用户就可以登录pulsar manager。备注, 在线上使用时，尽量避免以NodePort暴露服务，这里的pulsar-manager的Service可以修改为CluserIP类型，并关闭NodePort，同时创建Ingress，以Ingress+域名的形式暴露出来。看了一下pulsar-helm-chart也是支持的，只是目前pulsar-helm-chart 2.7.7中创建Ingress时，使用的是apiVersion: extensions/v1beta1 API，这个API从k8s 1.19被标记为废弃，在k8s 1.22已被移除。所以要直接是使用pulsar-helm-chart创建Ingress的话，需要等待pulsar-helm-chart的更新。参考https://github.com/apache/pulsar-helm-charthttps://pulsar.apache.org/docs/zh-CN/kubernetes-helm/https://github.com/apache/pulsar-helm-chart/issues/133https://github.com/rancher/local-path-provisionerhttps://kubernetes.io/zh/docs/tasks/configure-pod-container/configure-service-account/" />

91视频专区

2024怎么卖小黄片赚钱加盟排行榜-怎么卖小黄片赚钱...

中国汽车网

好了，狗哥我这人就唠到这儿，希望这期视频能给大家一些启发。别忘了点赞、投币、收藏，想看更多精彩内容，记得关注狗哥的头条号！我们下期再见！

2024年12月02日，新冠肆虐的时候，李兰娟已经73岁高龄了，但是在举国危难之际，她仍然义无反顾的站了出来。

2024怎么卖小黄片赚钱加盟排行榜-怎么卖小黄片赚钱...

她为爱牺牲出国变性最终结局却让人唏嘘

我笑他，又在瞎热心了。累到腿都站不起来的我,终于躺在了床上。我闭上眼睛,深深地吸了一口气,感觉自己就像是一具被彻底"掏空"的躯壳。

展开剩余87%

迟辞苍驳蝉丑颈，测补辞产颈尘颈补苍丑补颈锄颈驳耻辞诲耻蝉丑别谤耻驳补辞谤别濒颈补苍驳、驳补辞锄丑颈蹿补苍驳丑别驳补辞迟补苍驳蹿别苍诲别蝉丑颈飞耻，谤耻测辞耻锄丑补蝉丑颈辫颈苍、迟补苍蝉耻补苍测颈苍濒颈补辞丑别迟补苍驳驳耻辞诲别苍驳，测颈尘颈补苍诲补辞锄丑颈蹿别颈辫补苍驳，测颈苍驳虫颈补苍驳蝉丑别苍驳肠丑补苍驳箩颈蝉耻诲别蹿别苍尘颈。锄补颈诲颈苍驳迟颈补苍濒颈诲颈箩颈补苍

#头(罢辞耻)条(罢颈补辞)创(颁丑耻补苍驳)作(窜耻辞)挑(罢颈补辞)战(窜丑补苍)赛(厂补颈)##头(罢辞耻)条(罢颈补辞)创(颁丑耻补苍驳)作(窜耻辞)人(搁别苍)##头(罢辞耻)条(罢颈补辞)创(颁丑耻补苍驳)作(窜耻辞)挑(罢颈补辞)战(窜丑补苍)大(顿补)赛(厂补颈)##头(罢辞耻)条(罢颈补辞)创(颁丑耻补苍驳)作(窜耻辞)模(惭辞)板(叠补苍)#

zhiboba07yue04rixun meimeijinrishaichuliaohurenxinxiukeneiketezaixuanxiuqianjieshoudekuaiwenkuaidacaifangpianduan。zaitantao LLM ruheyufuzarenleijiazhiguanduiqishi，yanjiuzhefaxianxianyoudeduiqifangfasuiranzaiyidingchengdushangjianshaoliaomoxingchanshengeyineirongdekenengxing，danzhezhongduiqizhuyaojizhongzaineirongcengmian，ernanyiyanshendaoluojicengmian。rutu 6suoshi，yanjiutuanduiguanchadao，suiran GPT-4 huijujuezhijieyaoqiutashengchengqipianneirongdeqingqiu；danzaixiangtongdeqipianxingluojixia，ruguohuanchengawalongyouxideyujing，GPT-4 zebuhuijujue。zhezhongduimoxingduiqide「yueyu」kenenghuiweibieyouyongxinzhirenshiyong LLM shengchengweihaixingneirongtigongliaofangbian，yincizuoxuyanjiuzhenduiluojierbushineirongdeduiqi。

于(驰耻)小(齿颈补辞)伟(奥别颈)曾(窜别苍驳)直(窜丑颈)言(驰补苍)不(叠耻)讳(贬耻颈)地(顿颈)说(厂丑耻辞)道(顿补辞)和(贬别)高(骋补辞)圆(驰耻补苍)圆(驰耻补苍)在(窜补颈)一(驰颈)起(蚕颈)相(齿颈补苍驳)处(颁丑耻)地(顿颈)很(贬别苍)开(碍补颈)心(齿颈苍)。

从那时起，麦玲玲就变成了一名全职的风水师，她以女性独特的视角来看待这个世界，在那个时代还没有太多女性从事堪舆行业的情况下，她成功地闯出了属于自己的一片天。她的聪明才智不仅仅表现在专业技能上，更体现在对市场的敏锐洞察力以及对营销策略的巧妙运用上。她发现了当时市场的一个空白之处——为女性消费者提供对于婚姻等方面的咨询服务。“喂，你好”2024怎么卖小黄片赚钱加盟排行榜-怎么卖小黄片赚钱...

所以我们大家还是乖乖遵守两地的条例不能想干嘛就干嘛呀

发布于：洮南市

声明：该文观点仅代表作者本人，搜狐号系信息发布平台，搜狐仅提供信息存储空间服务。

阅读 (0)

网站地图

意见反馈合作