In our ongoing efforts to provide our customers with a best-in-class log management solution, we’re excited to launch improvements to our Agent. This new, robust version (3.4) provides significant performance improvements for our customers. Specifically, we focused on two main areas of the Agent: error handling and endurance.
Error Handling
One area of focus for Agent 3.4 is our ability to better handle error states and ensure you are optimizing performance. Specifically, we wanted to eliminate noisy error logs as well as give customers more control over their retry files. With Agent 3.4 we’ve implemented:
From our partners:
- Longer backoff for HTTP 5XX errors
- Immediate exiting on HTTP 4XX errors
- Recovery on notify errors
- Configurable retry file locations
- Maximum retry file sizes
- Additional retry metrics
- Compressed retry files
Endurance
We also focused our efforts on improving the Agent’s endurance. Specifically, we implemented:
- HTTP 2.0 support
- Concurrent uploads
The agent was enhanced to support tracking non-contiguous offsets, which allowed for concurrent uploads to be re-enabled without sacrificing stateful lookback.
Performance Data
As excited as we are for customers to try out this new Agent version and see these improvements for themselves, we know that we are only as good as the data we measure. Because of the focus on performance, we decided to share some of the data and metrics from our own internal testing. To help you understand the nature of this data, I have included a helpful glossary in the Appendix below.
Benchmark Testing
The goal of benchmark testing is to simulate the Agent under expected conditions. We ran benchmark tests on an EC2 instance in the us-west-1 region with the following specifications:
Instance type
t2.large
Architecture
x86_64
Processor
Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GH
Number of core
2
Memory
8 GB
Storage
EBS
And here were the results of the test:
File System Metrics
Total Time (sec)
23
Total Lines
4166932
Line Rate (lines/sec)
181170.95652173914
Total Size (bytes)
52245566.0
File Rate (bytes/sec)
2271546.347826087
Ingestion Metrics
Total Time (sec)
22
Total Size (bytes)
310600357.0
Total number of Samples
149
Average Request Duration (sec)
1.0604503484161074
Average Request Size (bytes)
2084566.1543624161
Endurance Testing
The goal here is to test the LogDNA Agent for an extended period under a pseudo-realistic workload. The following endurance test was run on an EC2 instance for 72 hours with the following specifications in the us-east-2 region:
Instance type
m5d.large
Architecture
x86_64
Processor
Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GH
Number of core
2
Memory
8 GB
Storage
EBS
Memory Consumption:
In these conditions, the logdna-agent consumed around 46 MB VmRSS, which measures how much RAM the process is actually using.
CPU Consumption:
Outside of the initial program start (which is reflected in the spike in the graph below, and is expected), the Agent consumed less than 2% of CPU throughout the entire endurance test.
In summary, Agent 3.4 performs well under heavy load for a long period of time and in error states that require some form of recovery. Please check out our documentation for further details. And as always, feel free to reach out to me with any questions: [email protected]
Appendix
Definitions:
File System Metrics
- Total Time: the time of the benchmark run in seconds.
- Total Lines: the total number of lines read off the file system by the agent, while the benchmark process was run.
- Line Rate: the number file lines per second read by the agent. Calculated as (total lines)/(total time)
- Total Size: the total amount of log data, in bytes, consumed by the agent.
- File Rate: the rate, in bytes per second, of log data being consumed by the agent. Calculated as (total size)/(total time)
Ingestion Metrics
- Total Time: same as defined above
- Total Size: The total amount of data sent via HTTP request to the Ingester.
- Total Number of Samples: The amount of samples taken by the metrics recorder during the benchmark run.
- Average Request Duration: the average duration of an HTTP call to the Ingester over the duration of the benchmark run. It is calculated as (total number of seconds spent sending HTTP requests to the ingester)/(total number of ingestion samples taken)
- Average Request Size: the average size of an HTTP request sent to the Ingester over the duration of the benchmark run. Calculated as (sum of all bytes sent to the ingester)/(total number of ingestion samples taken)
Memory Metrics
Max Process Virtual Memory: the maximum process virtual memory observed throughout the benchmark process run.
By Albert Feng
Source logDNA
For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!
Our humans need coffee too! Your support is highly appreciated, thank you!
