feat: add raw format and rawLogKey support for Fluent Bit Kafka

[Vaibhav-C-S]

What this PR does / why we need it:

Adds support for Fluent Bit Kafka raw output mode in fluent-operator.

Fluent Bit's Kafka output supports Format raw together with Raw_Log_Key, where the value of the configured record key is sent as the Kafka message payload. However, fluent-operator's Kafka output spec did not expose rawLogKey, and the documented format options did not include raw.

This PR adds support for:

• kafka.format: raw
• kafka.rawLogKey

Which issue(s) this PR fixes:

Fixes #1878

Does this PR introduced a user-facing change?

Added support for Fluent Bit Kafka raw output mode, allowing users to configure `format: raw` with `rawLogKey` to send a selected record field as the Kafka message payload.

Additional documentation, usage docs, etc.:

Testing

Testing

• Ran make test
• Tested locally on Minikube with Fluent Operator and Kafka
• Verified ClusterOutput accepts kafka.rawLogKey
• Verified generated Fluent Bit config contains

[Output]
    Name    kafka
    Match    dummy.*
    Format    raw
    Raw_Log_Key    log
    Brokers    kafka.kafka-test.svc.cluster.local:9092
    Topics    test

Message received at kafka consumer:
Input

[Input]
    Name    dummy
    Tag    dummy.test
    Dummy    {"log":"hello raw kafka from fluent-operator"}

Message received at kafka consumer

hello raw kafka from fluent-operator

[Copilot]

Pull request overview

This PR extends Fluent Operator’s Fluent Bit Kafka output plugin surface area to support Fluent Bit’s raw output mode, enabling users to send a selected record field as the Kafka message payload via Raw_Log_Key.

Changes:

• Add rawLogKey to the Kafka output plugin API and render it as Raw_Log_Key in generated Fluent Bit config.
• Document format: raw and rawLogKey in the Kafka output plugin docs.
• Regenerate CRDs and bundled manifests/Helm CRDs to include the new field and updated format description.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
manifests/setup/setup.yaml	Regenerated install manifest to include raw in format description and add rawLogKey schema/docs.
docs/plugins/fluentbit/output/kafka.md	Document format: raw and rawLogKey in the Kafka output plugin doc table.
config/crd/bases/fluentbit.fluent.io_outputs.yaml	Regenerated CRD base for Output to include rawLogKey and updated format description.
config/crd/bases/fluentbit.fluent.io_clusteroutputs.yaml	Regenerated CRD base for ClusterOutput to include rawLogKey and updated format description.
charts/fluent-operator/crds/fluentbit.fluent.io_outputs.yaml	Regenerated Helm CRD for Output with rawLogKey and updated format description.
charts/fluent-operator/crds/fluentbit.fluent.io_clusteroutputs.yaml	Regenerated Helm CRD for ClusterOutput with rawLogKey and updated format description.
charts/fluent-operator-fluent-bit-crds/templates/fluentbit.fluent.io_outputs.yaml	Regenerated Fluent Bit CRD chart template for Output with rawLogKey and updated format description.
charts/fluent-operator-fluent-bit-crds/templates/fluentbit.fluent.io_clusteroutputs.yaml	Regenerated Fluent Bit CRD chart template for ClusterOutput with rawLogKey and updated format description.
apis/fluentbit/v1alpha2/plugins/output/kafka_types.go	Add RawLogKey to the Kafka output API and emit Raw_Log_Key in Params.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

[Vaibhav-C-S]

hi @joshuabaird , can you please review this pr in your free time.

[joshuabaird]

@Vaibhav-C-S Hi! Thanks for the PR. Could you please review Copilot's comments?

[Vaibhav-C-S]

yes @joshuabaird , addressing

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

[Vaibhav-C-S]

Hi @joshuabaird , i have addressed all the review comments from co-pilot, can you check in your free time?

[joshuabaird]

Thanks for the iterations here — the raw/rawLogKey support, the early Params() guard, and the new test coverage all look good.

One concern before this merges: the new +kubebuilder:validation:Enum:=json;msgpack;raw marker on format is a backward-incompatible change to a pre-existing field.

format was previously a free-form string. Fluent Bit's Kafka output actually supports more values than the three listed — per the current docs it also accepts gelf, avro, otlp_json, and otlp_proto. With this enum in place:

• Existing users who set format: gelf (or avro/otlp) will have their Output/ClusterOutput rejected by the API server after the CRD upgrade.
• Those valid formats can no longer be configured going forward.

The Copilot suggestion that prompted the enum assumed json/msgpack/raw were the complete set, which isn't the case.

Recommendation: drop the enum and keep format a free-form string, matching upstream behavior. This PR's goal — adding raw + rawLogKey — doesn't require it, and the runtime guard you added (error when format: raw and rawLogKey is empty) already catches the one footgun that matters. This avoids any breaking change.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

[Vaibhav-C-S]

@joshuabaird Thanks for the catch, I do agree adding enum validation would be a breaking change. Made the code required. Thank you.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.