1.60.1

Instrumentations

OpenTelemetry instrumentation

• 🐛 Restore OpenTelemetry instrumentation library support. (#10738 - @bric3)

1.60.0

Known Bugs

Warning

This version broke the OpenTelemetry instrumentation library support.
If you use this feature, please upgrade to the latest version.

Components

Application Security Management (WAF)

• 🐛 Forbidden attempt to set different raw URI for given request context - issue #10700 (#10709 - @jandro996)
• ✨🧪 Set sampling decision maker for AI Guard (#10702 - @smola)
• 🐛 Always collect response headers in span when AppSec is enabled (#10648 - @jandro996)
• ✨ Force manual keep of traces for AI Guard (#10625 - @smola)
• 🐛 Optimize AppSecRequestContext numeric conversion to eliminate exceptions on hot path (#10520 - @jandro996)
• ✨ Improve API Security sampling when tracers lack HTTP routes(#10424 - @jandro996)
• ✨ HTTP Endpoint Inference for Trace Resource Renaming (#10393 - @jandro996)

Configuration

• ✨ Add DD_OTLP_METRICS_COMPRESSION setting and map OTel env-vars to it (#10680 - @mcculls)

Continuous Integration Visibility

• 🐛 Fix usage of deprecated field in AndroidGradleUtils (#10710 - @daniel-mohedano)
• 🧹 Refactor execution policies (#10676 - @daniel-mohedano)
• ✨ Add display name as parameter in JUnit5 dynamic tests (#10649 - @daniel-mohedano)
• ✨ Avoid computing merge base for PR info in DDCI legacy mode (#10624 - @daniel-mohedano)
• ✨⚡ Saving buildGitInfo lambda into a member variable (#10615 - @dougqh)
• ✨ Implement early exit for retry mechanisms (#10604 - @daniel-mohedano)
• 🐛 Validate path traversals in repo root (#10568 - @daniel-mohedano)
• ✨ Introduce pagination support for known tests API (#10536 - @daniel-mohedano)
• ✨ Create final status tag on test events (#10528 - @daniel-mohedano)
• ✨ Fix Job URL format for Github Actions (#10514 - @daniel-mohedano)
• ✨ Improve git security settings (#10486 - @daniel-mohedano)

Crash Tracking

• ✨ Preserve the function relative address when parsing a crash (#10544 - @amarziali)
• ✨ Add J9/OpenJ9 crash tracking support (#10496 - @jbachorik)
• ✨ Improve crashtracking payload and add build_id and relative address (#10469 - @amarziali)

Data Streams Monitoring

• 🐛 Fixed kafka produce extractor error (#10610 - @kr-igor)

Database Monitoring

• 🐛 Write base hash to current span when injected in DBM comment (#10450 - @vandonr)

Dynamic Instrumentation

• 🐛 Fix race condition for Failed Test Replay (#10679 - @daniel-mohedano)
• 🐛 Fix Windows path parsing in JarScanner (#10664 - @tylfin)
• ✨ Refactor to store samplers inside probes (#10605 - @jpbempel)
• 🐛 Fix SymDB jar processing (#10592 - @jpbempel)
• ✨ Send debugger logs to DEBUGGER track (#10585 - @jpbempel)
• 🐛 Fix Capture Expression on primitives (#10549 - @jpbempel)
• 🐛 Add rate limit on span decoration probe errors (#10529 - @jpbempel)
• 🐛 Fix method parameters JVM bug (#10521 - @jpbempel)
• 🐛 Fix Capture Expressions support for multi-probes (#10519 - @jpbempel)
• 🐛 Fix Exception Replay in Lambda (#10505 - @jpbempel)

Profiling

• ✨ Bump aircompressor to 2.0.3 (#10673 - @jbachorik)
• 🐛 Disable jdk.CPUTimeSample when ddprof handles CPU profiling (#10634 - @jbachorik)
• ✨ Add JFR scrubbing before profile upload (#10577 - @jbachorik)

Trace context propagation

• ✨⚡ Changing traceSource to volatile rather than an AtomicInteger (#10614 - @dougqh)

Tracer core

• ✨ Add manual service name source (#10704 - @amarziali)
• 🐛 Do not remove the version tag if manually set (#10703 - @amarziali)
• 🧹 Upgrade to jctools 4.0.6 (#10691 - @amarziali)
• ✨ Refine service name sources for configuration options (#10658 - @amarziali)
• ✨ Add service name source to trace stats (#10653 - @amarziali)
• ✨⚡ Creating HashingUtils as alternative to Objects.hash (#10628 - @dougqh)
• ✨ Track service name source (#10607 - @amarziali)
• 🐛 Make MemFDUnixWriterFFM compatible with early SegmentAllocator jdk implementations (#10602 - @amarziali)
• ⚡🧹 Remove use of AtomicReferenceArray in RadixTreeCache (#10600 - @mcculls)
• ✨ Propagate response body in case of non 200 status code. (#10593 - @AlexeyKuznetsov-DD)
• ✨ Update jnr-unixsocket dependency to 0.38.24 (#10565 - @mcculls)
• 🐛 Avoid potential ConcurrentModificationException in TunnelingJdkSocket (#10564 - @mcculls)
• ✨ Add primitive versions of setMetric to DDSpanContext (#10516 - @dougqh)
• ✨⚡ Reuse SpanKind Entry in ClientDecorator (#10503 - @dougqh)
• ✨⚡ Reducing Allocation from GitInfo (#10498 - @dougqh)
• ✨⚡ Caching the primaryInstrumentationName (#10477 - @dougqh)
• ✨⚡ Adds methods for setting TagMap.Entry directly on AgentSpan/DDSpan (#10472 - @dougqh)
• 🐛 Remove RadixTreeCache for int status and protect httpStatus short cache (#10236 - @amarziali)

Tracer internal logging

• ✨ Adding Logging for JMS ITERATION Scope (#10632 - @mhlidd)

Instrumentations

Apache Spark instrumentation

• ✨ Spark instrumentation: capture emr step id when spark runs as EMR step (#10670 - @aboitreaud)
• ✨ Initial spark launcher instrumentation (#10629 - @aboitreaud)

AWS Lambda instrumentation

• 🐛 Fix Exception Replay in Lambda (#10505 - @jpbempel)

Core Java language instrumentation

• 🐛 Fixed issue with recursive calls which can cause StackOverflowError. (#10535 - @AlexeyKuznetsov-DD)
• ✨⚡ Reuse TagMap.Entry objects in BaseDecorator (#10501 - @dougqh)

JDBC instrumentation

• 🐛 DbInfo Thread Safety (#10517 - @dougqh)

JMS instrumentation

• ✨ Adding Logging for JMS ITERATION Scope (#10632 - @mhlidd)

Netty instrumentation

• 🐛 Track server context in netty4 instrumentation (#10556 - @mcculls)
• 🐛 Track server context in netty3 instrumentation (#10555 - @mcculls)

OpenTelemetry instrumentation

• ✨ Add DD_OTLP_METRICS_COMPRESSION setting and map OTel env-vars to it (#10680 - @mcculls)
• 💡 Implement OpenTelemetry Observable Metrics API (#10631 - @mcculls)
• 💡 Adjust OpenTelemetry metrics temporality according to global preference and instrument type (#10545 - @mcculls)
• 💡 Add support for otel.java.metrics.cardinality.limit system property (#10541 - @mcculls)
• 💡 Implement OpenTelemetry meter storage and aggregations (#10534 - @mcculls)

Play Framework instrumentation

• 🐛 Combine play spans with surrounding framework context before attaching them (#10554 - @mcculls)

Servlet instrumentation

• 🐛 Combine dispatch span with request context in generic Servlet3 advice (#10557 - @mcculls)
• ✨⚡ Reuse TagMap.Entry objects in BaseDecorator (#10501 - @dougqh)
• ✨⚡ Extracting span kind & language into static TagMap.Entry objects (#10495 - @dougqh)
• ✨⚡ Caching the primaryInstrumentationName (#10477 - @dougqh)

Synapse Instrumentation

• ✨ Fix Synapse passthru spans finishing before status code is available (#10478 - @ValentinZakharov)

1.59.0

Breaking Changes

Note

Manually overriding a sampling decision now always overrides the decision maker tag for future distributed trace propagation.

Components

Application Security Management (WAF)

• ✨🧪 Add support for content parts and image URLs in AI Guard (#10449 - @smola)
• ✨ Analyze okhttp client redirections as separated request (#10252 - @manuel-alvarez-alvarez)

Continuous Integration Visibility

• ✨ Use component as framework tag in manual API (#10313 - @daniel-mohedano)

Crash Tracking

• ✨ Strip temp suffix from known library names (#10335 - @amarziali)

Data Streams Monitoring

• 🐛 Avoid NPE on DSM context extraction (#10491 - @amarziali)
• 🐛 Fix DSM queue names with Kafka Connect IBM MQ connectors (#10318 - @johannbotha)
• ✨ Initial version of transactions tracking implementation for DSM (#9899 - @kr-igor)

Dynamic Instrumentation

• 🐛 Fix SymDB startup (#10493 - @jpbempel)
• ✨ Enable Code Origin by default for JDK21+ (#10468 - @jpbempel)
• 🐛 Avoid throwing exception for invalid modifiers (#10458 - @jpbempel)
• 🐛 Fix SymDB when processing corrupted jars (#10453 - @jpbempel)
• ✨ Add support of Kotlin's SourceDebugExtension (#10448 - @jpbempel)
• 🐛 Fix debugger start when endpoints aren't available (#10401 - @jpbempel)
• 🐛 Fix NullPointerException when scope failure (#10333 - @jpbempel)
• 🐛 Fix exceptions raised from Collection values (#10322 - @jpbempel)
• 🐛 fix NullPointerException in ClassesToRetransformFinder (#10316 - @jpbempel)
• 🐛 Fix ArrayOutOfBoundsException in ER sanity checks (#10314 - @jpbempel)
• 🐛 fix Capture Expressions for line probes (#10303 - @jpbempel)

Library Injection

• ✨ Enable websocket instrumentation by default (#10437 - @tlhunter)

ML Observability (LLMObs)

• 🐛 Fix LLMObs trace batching and payload size telemetry (#10395 - @ygree)
• 🐛 Avoid sending empty payloads to the LLMObs endpoint (#10372 - @ygree)
• 💡 Add openai-java v3.0+ instrumentation (#9959 - @ygree)

Metrics

• 🧹 Extract metrics from communication (#10399 - @PerfectSlayer)

OpenFeature

• 🐛 Add test case for OF.7: empty targeting key compliance (#10445 - @leoromanovsky)
• 🐛 Bump OpenFeature SDK to 1.20.1 (#10438 - @leoromanovsky)
• 🐛 Fix ISO 8601 date parsing to support variable precision (#10419 - @leoromanovsky)

Profiling

• ✨ Bump ddprof to 1.37.0 (#10487 - @jbachorik)
  (see DataDog/java-profiler@v_1.34.4...v_1.37.0 for details)
• 🐛 Should not access the thread states of none Java threads by @zhengyu123 in DataDog/java-profiler#346
• 🐛 Add alignment checks to NativeFunc mark methods by @jbachorik in DataDog/java-profiler#348
• ✨ Log profiler flare messages at WARN level (#10444 - @jbachorik)
• 🐛 Allow running wallclock profiler even without tracer (#10298 - @jbachorik)

Trace context propagation

• 🐛⚠️ Always record force keep decisions for future distributed trace propagation (#10308 - @mcculls)

Tracer core

• ✨ Add process tags related to service naming (#10480 - @amarziali)
• ✨⚡ Avoid primitive boxing in v0.5 serializer (#10471 - @dougqh)
• ✨ Add safety check for TargetSystem encoding limits (#10451 - @amarziali)
• ✨ Compute basehash without allocating a stringbuilder (#10411 - @vandonr)
• ✨⚡ Avoid primitive boxing in v0.4 serializer (#10408 - @dougqh)
• ✨ Introduce @AppliesOn to override advices InstrumenterModule target system (#10404 - @amarziali)
• ✨⚡ Introducing TagMap EntryReader (#10339 - @dougqh)
• 🐛 Ledger.reset should reset containsRemoval status (#10338 - @dougqh)
• ✨ Service discovery: defer FFM methodhandles resolutions (#10312 - @amarziali)
• ✨ Use pure FFM to write on service discovery memfd from java 22 (#10295 - @amarziali)
• ✨⚡ Enabling optimized map by default (#10016 - @dougqh)
• ✨ Replace JCTools queues with VarHandle-based implementations for Java 25+ (#9896 - @amarziali)

Tracer internal logging

• ✨ Support DD_TRACE_LOG_LEVEL (#10441 - @mcculls)

Instrumentations

Eclipse Vert.x instrumentation

• 🐛 Have Vertx 4.0 prioritize path and fallback to name (#10382 - @mhlidd)

GraphQL instrumentation

• 🐛 Add Custom Exception Handler to Unwrap CompletionException for GraphQL Instrumentations (#10389 - @mhlidd)

Kafka instrumentation

• 🐛 Fix missing consumer_group and topic tags in DSM Kafka 3.8+ metrics (#10464 - @johannbotha)
• ✨ Use WithTypeStructure to check declaresField (#10307 - @amarziali)

WebSocket Instrumentation

• ✨ Enable websocket instrumentation by default (#10437 - @tlhunter)

All other instrumentations

• 🧹 Move pure context tracking instrumenter to InstrumenterModule.ContexTracking (#10425 - @amarziali)
• 🐛 Avoid duplicate headers when injecting on java.net http client (#10315 - @amarziali)
• ✨ CICS tracing (#10301 - @deejgregor - thanks for the contribution!)
• 🐛 Avoid null store put on Ignite instrumentation (#10291 - @amarziali)
• 💡 Add openai-java v3.0+ instrumentation (#9959 - @ygree)

1.58.2

Known Bugs

Warning

The a bug in the profiler can cause crashes on libmusl. If running on libmusl based distribution, consider not upgrading or, if absolutely unavoidable, use -Ddd.profiling.ddprof.enabled=false to fall back to JFR-only profiling to avoid the possibility of crash.

Components

ML Observability (LLMObs)

• 🐛 Fix LLMObs trace batching and payload size telemetry (#10417 - @ygree)
• 🐛 Avoid sending empty payloads to the LLMObs endpoint (#10410 - @ygree)

Profiling

• ✨ Bump java profiler to 1.36.1 (#10428 - @ygree)

1.58.1

Components

Continuous Integration Visibility

• ✨ Use component as framework tag in manual API (#10374 - @daniel-mohedano)

Instrumentations

All other instrumentations

• 🐛 Add InstrumenterModule.Tracing for Aerospike (#10373 - @amarziali)

1.58.0

Known Bugs

Warning

The Aerospike instrumentation does not apply in this dd-java-agent release. If you are instrumenting Aerospike, please upgrade to the latest version.

Breaking Changes

Note

We now limit the number of elements processed in Mongo documents (#10277). This only affects unusually long Mongo resource names derived from documents with more than 256 list elements.

Components

Application Security Management (WAF)

• 🐛 Fix synchronization of ASM_DD Product listeners (#10226 - @jandro996)
• ✨ Add Security Response ID (#10133 - @jandro996)

Data Streams Monitoring

• 🐛 Remove high cardinality tags from Data Streams hashes (#10287 - @piochelepiotr)

Database Monitoring

• ✨ Add support for DBM comment injection with MongoDB (#9589 - @Yoone)

Dynamic Instrumentation

• 🐛⚡ Batch re-transformations when there are a high number of inner classes (#10289 - @jpbempel)
• ✨ Add support for Agrona collections (#10235 - @jpbempel)
• ✨ Enable Code Origin by default for JDK25+ (#10234 - @jpbempel)

Tracer core

• ✨ Upgrade to byte-buddy 1.18.3 (#10275 - @mcculls)
• ✨ Force early classloading for PublishState (#10273 - @amarziali)
• ✨⚡ Support creating AOT caches for Spring apps with custom trace interceptors (#10272 - @mcculls)
• 🐛 Ensure the class name is valid when enhancing with SerialVersionUIDAdder (#10270 - @amarziali)
• 🐛 Silence non-critical PayloadTagsProcessor parsing exception alert (#10263 - @ygree)

Tracer internal logging

• ✨ Support DD_LOG_FORMAT_JSON (#10223 - @mcculls)
• 💡 Add App Logs Collection feature (#10156 - @jpbempel)

Instrumentations

Apache Spark instrumentation

• ✨ Enable Spark Plan metadata parsing by default (#10169 - @charlesmyu)

Couchbase instrumentation

• 🐛 Multi-tracing support for Couchbase 3.2+ (#10147 - @ValentinZakharov)

JMS instrumentation

• 🐛 Do not break the JMS instrumentation when unable to access a property (#10255 - @amarziali)

MongoDB instrumentation

• 🐛⚠️ Add a limit to the number of elements we process in a Mongo document (#10277 - @vandonr)
• ✨ Add support for DBM comment injection with MongoDB (#9589 - @Yoone)

Servlet instrumentation

• 🐛 Protect jsp decorator from NPE (#10250 - @amarziali)

Spring instrumentation

• 🐛 Avoid NPE when inspecting spring messaging headers (#10254 - @amarziali)

All other instrumentations

• 🐛 Avoid NPE in tomcat classloading instrumentation (#10256 - @amarziali)

1.57.0

Components

Application Security Management (IAST)

• 🐛 Fix IllegalFormatConversionException StringModuleImpl#onStringFormat (#9907 - @jandro996)

Application Security Management (WAF)

• 🐛 Fix API Security sampling in standalone mode (#10165 - @jandro996)
• ✨ AI Guard - Ensure messages are not modified before span serialization (#10116 - @manuel-alvarez-alvarez)

Build & Tooling

• ✨ Remove native libraries for unsupported environment (#10079 - @PerfectSlayer)

Configuration

• 🐛 Using stable config when a security manager is enabled should not prevent tracer startup (#10006 - @amarziali)

Continuous Integration Visibility

• ✨ Add support for Weaver Framework v0.11 (#10012 - @daniel-mohedano)

Crash Tracking

• 🐛 Don't send crash-ping to error tracking if not enabled (#10170 - @amarziali)
• ✨ Improve OOM parsing for Zulu8 (#10038 - @amarziali)
• ✨ Store some crashtracking options on initialisation (#10036 - @amarziali)
• ✨ Dual ship to error tracking (#10000 - @amarziali)

Data Streams Monitoring

• 🐛 Fix Kafka consumer instrumentation (#10100 - @piochelepiotr)

Dynamic Instrumentation

• 🐛 Fix metric expressions with @return (#10178 - @jpbempel)
• 🧹 fix method reference in comment (#10101 - @andreimatei)
• ✨ Make SymbolDB stand alone (#10064 - @jpbempel)
• 🐛 Make Code Origin fingerprint map concurrent (#10043 - @jpbempel)
• 🐛 Disable CodeOrigin by default (#10027 - @jpbempel)
• 🐛⚡ Avoid expensive Class.getMethod call in code-origin advice (#10019 - @mcculls)

Metrics

• ✨⚡ Avoid getTag overhead in ConflatingMetricsAggregator (#10070 - @amarziali)
• 🐛 Only enable client side stats if the host agent is at least 7.65.0 (#10041 - @bric3)

Platform

• ✨ Improve support for virtual thread detection (#9852 - @PerfectSlayer)

Profiling

• ✨ Bump ddprof-java to 1.34.4 (#10097 - @jbachorik)
• 🐛 Fix deadlock in dd-task-scheduler (#10096 - @rkennke)

Realtime User Monitoring

• 🐛 Filter setContentType headers (#10150 - @amarziali)
• 🐛 Prevent any Content-Length header from being set upon injection (#10081 - @amarziali)

Telemetry

• ✨ Don't send OutOfMemoryError to the telemetry (#10211 - @amarziali)

Trace context propagation

• ✨ Add context tracking support for virtual threads (#10040 - @PerfectSlayer)

Tracer core

• ✨ Don't send IO failures to telemetry logs (#10210 - @amarziali)
• 🐛 Avoid NPE on BaseDecorator (#10209 - @amarziali)
• ✨⚡ Support aot_training mode to be used when creating AOT caches (#10166 - @mcculls)
• ✨⚡ Optimize TagMap lookup & insertion for String literals (#10138 - @dougqh)
• 🐛 Remove colons from process tags values (#10109 - @vandonr)
• 🐛 Fix field-injection of ForkJoinTask on Java 25 (#10084 - @mcculls)
• ✨⚡ Use ClassLoaderValues to track class-loader state (#10075 - @mcculls)
• ✨⚡ Add primitive setters to DDSpanContext to take advantage of TagMap (#10018 - @dougqh)
• 🐛 Safely handle Files.exist on discovery and config when a Security Manager is present (#10009 - @amarziali)
• ✨⚡ Avoid SpanBuilder allocation for startSpan (#9998 - @dougqh)
• ✨ Add request-id to lambda start/end invocation calls (#9991 - @ojproductions)
• ✨⚡ Use ClassLoaderIndex to index class-loader related information (#9980 - @mcculls)
• ⚡🧹 Use ClassNameFilter to track uninteresting classes (#9976 - @mcculls)
• ✨⚡ Reduce PendingTrace Lock Contention (#9932 - @dougqh)
• ✨ Remove dependency on memfd_create for service discovery (#9913 - @raphaelgavache)

Instrumentations

Apache Spark instrumentation

• 🐛 Support Spark 4.1 by removing mention to StreamExecution class (#10228 - @aboitreaud)
• ✨ Duplicate service name in meta so it's not truncated in spark integration (#10063 - @aboitreaud)
• ✨ Add dd_tags into djm.tags spark.application span attribute (#10004 - @pawel-big-lebowski)

AWS SDK instrumentation

• 💡 Add inject Datadog attribute config (#9943 - @ojproductions)

Core Java language instrumentation

• ✨ Add context tracking support for virtual threads (#10040 - @PerfectSlayer)

JDBC instrumentation

• 🐛 Log injection issues in JDBC only once + promote to WARN (#10214 - @vandonr)
• 💡 Add a feature flag to disable extra calls to the DB in JDBC instrumentation (#9774 - @vandonr)

Kafka instrumentation

• 🐛 Fix Kafka consumer instrumentation (#10100 - @piochelepiotr)
• 🐛 Extract trace context from Kafka producer record headers (#10020 - @PlugaruT - thanks for the contribution!)

Servlet instrumentation

• 🐛 Avoid NPE when enumerating servlet headers (#10213 - @amarziali)

Spring instrumentation

• 🐛 Make spring-webflux instrumentation compatible with 7.0.0 (#10110 - @amarziali)

All other instrumentations

• 🐛 Avoid NPE on QuartzDecorator (#10181 - @amarziali)
• 🐛 Avoid NPE when setting resource with null path (#10180 - @amarziali)
• 🐛 Fix VerifyError when instrumenting org.apache.synapse.transport.passthru.SourceHandler (#10154 - @mcculls)
• 🐛 Ensure compatibility with jboss modules 2.2 (#10111 - @amarziali)

1.56.3

Components

Profiling

• ✨ Bump ddprof-java to 1.34.4 (#10153 - @jbachorik)
  (see DataDog/java-profiler@v_1.34.0...v_1.34.4 for details)

Instrumentations

JDBC instrumentation

• 💡 Add a feature flag to disable extra calls to the DB in JDBC instrumentation (#10161 - @vandonr)

1.56.2

Components

Data Streams Monitoring

• 🐛 Fix Kafka consumer instrumentation (#10102 - @piochelepiotr)

Instrumentations

Kafka instrumentation

• 🐛 Fix Kafka consumer instrumentation (#10102 - @piochelepiotr)

1.56.1

Components

Dynamic Instrumentation

• 🐛 Disable CodeOrigin by default (#10029 - @mcculls)