Revisions of librdkafka
buildservice-autocommit
accepted
request 1127461
from
Dirk Mueller (dirkmueller)
(revision 41)
baserev update by copy to link target
Dirk Mueller (dirkmueller)
committed
(revision 40)
- update to 2.3.0: * Partial support of topic identifiers. Topic identifiers in metadata response available through the new `rd_kafka_DescribeTopics` function * KIP-117 Add support for AdminAPI `DescribeCluster()` and `DescribeTopics()` * Return authorized operations in Describe Responses. * KIP-580: Added Exponential Backoff mechanism for retriable requests with `retry.backoff.ms` as minimum backoff and `retry.backoff.max.ms` as the maximum backoff, with 20% jitter (#4422). * Fixed ListConsumerGroupOffsets not fetching offsets for all the topics in a group with Apache Kafka version below 2.4.0. * Add missing destroy that leads to leaking partition structure memory when there are partition leader changes and a stale leader epoch is received (#4429). * Fix a segmentation fault when closing a consumer using the cooperative-sticky assignor before the first assignment * Fix for insufficient buffer allocation when allocating rack information (@wolfchimneyrock, #4449). * Fix for infinite loop of OffsetForLeaderEpoch requests on quick leader changes. (#4433). * Fix for stored offsets not being committed if they lacked the leader epoch (#4442). * Upgrade OpenSSL to v3.0.11 (while building from source) with various security fixes, check the release notes * Fix to ensure permanent errors during offset validation continue being retried and don't cause an offset reset (#4447). * Fix to ensure max.poll.interval.ms is reset when rd_kafka_poll is called with consume_cb (#4431).
buildservice-autocommit
accepted
request 1087789
from
Dirk Mueller (dirkmueller)
(revision 39)
baserev update by copy to link target
Dirk Mueller (dirkmueller)
committed
(revision 38)
Dirk Mueller (dirkmueller)
committed
(revision 37)
Dirk Mueller (dirkmueller)
committed
(revision 36)
Dirk Mueller (dirkmueller)
committed
(revision 35)
- update to 2.1.1: * Avoid duplicate messages when a fetch response is received * in the middle of an offset validation request * Fix segmentation fault when subscribing to a non-existent topic and calling `rd_kafka_message_leader_epoch()` on the polled `rkmessage` * Fix a segmentation fault when fetching from follower and the partition lease expires while waiting for the result of a list offsets operation * Fix documentation for the admin request timeout, incorrectly stating -1 for infinite * timeout. That timeout can't be infinite. * Fix CMake pkg-config cURL require and use * pkg-config `Requires.private` field * Fixes certain cases where polling would not keep the consumer * in the group or make it rejoin it * Fix to the C++ set_leader_epoch method of TopicPartitionImpl, * that wasn't storing the passed value * Duplicate messages can be emitted when a fetch response is received in the middle of an offset validation request. Solved by avoiding a restart from last application offset when offset validation succeeds. * When fetching from follower, if the partition lease expires after 5 minutes, and a list offsets operation was requested to retrieve the earliest or latest offset, it resulted in segmentation fault. This was fixed by allowing threads different from the main one to call the `rd_kafka_toppar_set_fetch_state` function, given they hold the lock on the `rktp`. * In v2.1.0, a bug was fixed which caused polling any queue to reset the `max.poll.interval.ms`.
buildservice-autocommit
accepted
request 1083981
from
Marcus Meissner (msmeissn)
(revision 34)
baserev update by copy to link target
Marcus Meissner (msmeissn)
accepted
request 1083355
from
Dirk Mueller (dirkmueller)
(revision 33)
- update to 2.1.0: * Allow fetchers to detect and handle log truncation (#4122). * Fix a reference count issue blocking the consumer from closing (#4187). * Fix a protocol issue with ListGroups API, where an extra * field was appended for API Versions greater than or equal to 3 (#4207). * Fix an issue with `max.poll.interval.ms`, where polling any queue would cause the timeout to be reset (#4176). * Fix seek partition timeout, was one thousand times lower than the passed value (#4230). * Fix multiple inconsistent behaviour in batch APIs during **pause** or **resume** operations (#4208). * Update lz4.c from upstream. Fixes CVE-2021-3520 * Upgrade OpenSSL to v3.0.8 with various security fixes * Added `rd_kafka_topic_partition_get_leader_epoch()` (and `set..()`). * A reference count issue was blocking the consumer from closing. * Fixed known issues related to Batch Consume APIs mentioned in v2.0.0 release notes. * Fixed `rd_kafka_consume_batch()` and `rd_kafka_consume_batch_queue()` intermittently updating `app_offset` and `store_offset` incorrectly when **pause** or **resume** was being used for a partition. * Fixed `rd_kafka_consume_batch()` and `rd_kafka_consume_batch_queue()` intermittently skipping offsets when **pause** or **resume** was being used for a partition.
buildservice-autocommit
accepted
request 1062038
from
Marcus Meissner (msmeissn)
(revision 32)
baserev update by copy to link target
Marcus Meissner (msmeissn)
accepted
request 1061868
from
Dirk Mueller (dirkmueller)
(revision 31)
- update to 2.0.2: * OffsetFetch Protocol Update (#3995). * Add Consumer Group operations to Admin API (started by @lesterfan, #3995). * Allow listing consumer groups per state (#3995). * Partially implemented: support for AlterConsumerGroupOffsets * OpenSSL 3.0.x support - the maximum bundled OpenSSL version is now 3.0.7 (previously 1.1.1q). * Fixes to the transactional and idempotent producer. * The introduction of OpenSSL 3.0.x in the self-contained librdkafka bundles changes the default set of available ciphers, in particular all obsolete or insecure ciphers and algorithms as listed in the OpenSSL legacy manual page are now disabled by default. Should you need to use any of these old ciphers you'll need to explicitly enable the `legacy` provider by configuring `ssl.providers=default,legacy` on the librdkafka client. OpenSSL 3.0.x deprecates the use of engines, which is being replaced by providers. As such librdkafka will emit a deprecation warning if `ssl.engine.location` is configured. OpenSSL providers may be configured with the new `ssl.providers` configuration property. The default value for `ssl.endpoint.identification.algorithm` has been changed from `none` (no hostname verification) to `https`, which enables broker hostname verification (to counter man-in-the-middle impersonation attacks) by default. To restore the previous behaviour, set `ssl.endpoint.identification.algorithm` to `none`. * The Consumer Batch APIs `rd_kafka_consume_batch()` and `rd_kafka_consume_batch_queue()` are not thread safe if `rkmessages_size` is greater than 1 and any of the **seek**, **pause**, **resume** or **rebalancing** operation is performed in parallel with any of the above APIs. Some of the messages might be lost, or erroneously returned to the application, in the above scenario. * It is strongly recommended to use the Consumer Batch APIs and the
buildservice-autocommit
accepted
request 1004569
from
Marcus Meissner (msmeissn)
(revision 30)
baserev update by copy to link target
Marcus Meissner (msmeissn)
accepted
request 1003041
from
Dirk Mueller (dirkmueller)
(revision 29)
- update to 1.9.2: * Added KIP-768 OUATHBEARER OIDC support (by @jliunyu, #3560) * Added KIP-140 Admin API ACL support (by @emasab, #2676) * Consumer: `rd_kafka_offsets_store()` (et.al) will now return an error for any partition that is not currently assigned (through `rd_kafka_*assign()`). This prevents a race condition where an application would store offsets after the assigned partitions had been revoked (which resets the stored offset), that could cause these old stored offsets to be committed later when the same partitions were assigned to this consumer again - effectively overwriting any committed offsets by any consumers that were assigned the same partitions previously. This would typically result in the offsets rewinding and messages to be reprocessed. As an extra effort to avoid this situation the stored offset is now also reset when partitions are assigned (through `rd_kafka_*assign()`). Applications that explicitly call `..offset*_store()` will now need to handle the case where `RD_KAFKA_RESP_ERR__STATE` is returned in the per-partition `.err` field - meaning the partition is no longer assigned to this consumer and the offset could not be stored for commit. * Improved producer queue scheduling. Fixes the performance regression introduced in v1.7.0 for some produce patterns. (#3538, #2912) * Windows: Added native Win32 IO/Queue scheduling. This removes the internal TCP loopback connections that were previously used for timely queue wakeups. * Added `socket.connection.setup.timeout.ms` (default 30s). The maximum time allowed for broker connection setups (TCP connection as well as SSL and SASL handshakes) is now limited to this value. This fixes the issue with stalled broker connections in the case of network or load balancer problems. The Java clients has an exponential backoff to this timeout which is
Marcus Meissner (msmeissn)
accepted
request 943868
from
Dirk Mueller (dirkmueller)
(revision 28)
- update to 1.8.2: * Added ssl.ca.pem to add CA certificate by PEM string * Upon quick repeated leader changes the transactional producer could receive an OUT_OF_ORDER_SEQUENCE error from the broker * The transactional producer could stall during a transaction if the transaction coordinator changed
Marcus Meissner (msmeissn)
accepted
request 925676
from
Dirk Mueller (dirkmueller)
(revision 27)
- update to 1.8.0: * Upgrade bundled zlib version from 1.2.8 to 1.2.11 in the `librdkafka.redist` NuGet package. The updated zlib version fixes CVEs: CVE-2016-9840, CVE-2016-9841, CVE-2016-9842, CVE-2016-9843 See https://github.com/edenhill/librdkafka/issues/2934 for more information. * librdkafka now uses [vcpkg](https://vcpkg.io/) for up-to-date Windows dependencies in the `librdkafka.redist` NuGet package: OpenSSL 1.1.1l, zlib 1.2.11, zstd 1.5.0. * The upstream dependency (OpenSSL, zstd, zlib) source archive checksums are now verified when building with `./configure --install-deps`. These builds are used by the librdkafka builds bundled with confluent-kafka-go, confluent-kafka-python and confluent-kafka-dotnet. * Producer `flush()` now overrides the `linger.ms` setting for the duration of the `flush()` call, effectively triggering immediate transmission of queued messages. (#3489) * Lots of bugfixes, see included CHANGELOG.md for details - build against system libraries rather than bundled ones - enable all features
Marcus Meissner (msmeissn)
accepted
request 897827
from
Dirk Mueller (dirkmueller)
(revision 26)
- update to 1.7.0: * [KIP-360](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820) - Improve reliability of transactional producer. Requires Apache Kafka 2.5 or later. * OpenSSL Engine support (`ssl.engine.location`) by @adinigam and @ajbarb. * Added `connections.max.idle.ms` to automatically close idle broker connections. This feature is disabled by default unless `bootstrap.servers` contains the string `azure` in which case the default is set to <4 minutes to improve connection reliability and circumvent limitations with the Azure load balancers (see #3109 for more information). * Bumped to OpenSSL 1.1.1k in binary librdkafka artifacts. * The binary librdkafka artifacts for Alpine are now using Alpine 3.12. OpenSSL 1.1.1k. * Improved static librdkafka Windows builds using MinGW (@neptoess, #3130). * The C++ `oauthbearer_token_refresh_cb()` was missing a `Handle *` argument that has now been added. This is a breaking change but the original function signature is considered a bug. This change only affects C++ OAuth developers. * [KIP-735](https://cwiki.apache.org/confluence/display/KAFKA/KIP-735%3A+Increase+default+consumer+session+timeout) The consumer `session.timeout.ms` default was changed from 10 to 45 seconds to make consumer groups more robust and less sensitive to temporary network and cluster issues. * Statistics: `consumer_lag` is now using the `committed_offset`, while the new `consumer_lag_stored` is using `stored_offset` (offset to be committed). This is more correct than the previous `consumer_lag` which was using either `committed_offset` or `app_offset` (last message passed to application). * Bugfixes
buildservice-autocommit
accepted
request 888420
from
Dirk Mueller (dirkmueller)
(revision 25)
baserev update by copy to link target
Dirk Mueller (dirkmueller)
committed
(revision 24)
- update to 1.6.1: * Fatal idempotent producer errors are now also fatal to the transactional producer. This is a necessary step to maintain data integrity prior to librdkafka supporting KIP-360. Applications should check any transactional API errors for the is_fatal flag and decommission the transactional producer if the flag is set. * The consumer error raised by `auto.offset.reset=error` now has error-code set to `ERR__AUTO_OFFSET_RESET` to allow an application to differentiate between auto offset resets and other consumer errors. * Admin API and transactional `send_offsets_to_transaction()` coordinator requests, such as TxnOffsetCommitRequest, could in rare cases be sent multiple times which could cause a crash. * `ssl.ca.location=probe` is now enabled by default on Mac OSX since the librdkafka-bundled OpenSSL might not have the same default CA search paths as the system or brew installed OpenSSL. Probing scans all known locations. * Fatal idempotent producer errors are now also fatal to the transactional producer. * The transactional producer could crash if the transaction failed while `send_offsets_to_transaction()` was called. * Group coordinator requests for transactional `send_offsets_to_transaction()` calls would leak memory if the underlying request was attempted to be sent after the transaction had failed. * When gradually producing to multiple partitions (resulting in multiple underlying AddPartitionsToTxnRequests) sub-sequent partitions could get stuck in pending state under certain conditions. These pending partitions would not send queued messages to the broker and eventually trigger message timeouts, failing the current transaction. This is now fixed. * Committing an empty transaction (no messages were produced and no offsets were sent) would previously raise a fatal error due to invalid state
buildservice-autocommit
accepted
request 858094
from
Dirk Mueller (dirkmueller)
(revision 23)
baserev update by copy to link target
Dirk Mueller (dirkmueller)
committed
(revision 22)
- update to 1.5.3: * Fix a use-after-free crash when certain coordinator requests were retried. * Consumer would not filter out messages for aborted transactions if the messages were compressed (#3020). * Consumer destroy without prior `close()` could hang in certain cgrp states (@gridaphobe, #3127). * Fix possible null dereference in `Message::errstr()` (#3140). * The `roundrobin` partition assignment strategy could get stuck in an endless loop or generate uneven assignments in case the group members had asymmetric subscriptions (e.g., c1 subscribes to t1,t2 while c2 subscribes to t2,t3). (#3159)
Displaying revisions 1 - 20 of 41