Cloudflare is a content delivery network (CDN) that enables organizations to accelerate, protect, and optimize their websites, applications, and APIs. It performs as a reverse proxy between the internet and the website’s servers, providing DDoS protection, caching, and load balancing. Organizations use Cloudflare to improve website performance, reduce egress costs, and enhance security.

A Cloudflare Log

Cloudflare logs are structured logs in JSON format. Here are some of the key fields that provide useful information.

• EdgeStartTimestamp - When Cloudflare received the request
• EdgeEndTimestamp - When Cloudflare finished processing the request
• ClientRequestQuery - The request parameters
• EdgeResponseStatus - HTTP status code: 200, 404, 500, etc.
• CacheStatus - The result of an attempt to serve a request from cache: HIT, MISS, EXPIRED, BYPASS
• OriginIP - The IP address of the web server
• OriginTLSVersion - TLS version of the connection to the origin server
• OriginResponseDurationMs - Response time in milliseconds of the origin server
• WAFAction - Web Application Firewall action, ALLOW, BLOCK, CHALLENGE
• BotScore - Indication score for bot detection
• ThreatScore - Threat intelligence score
• ASN - Autonomous System Number of the client IP
• ClientSSLProtocol - SSL/TLS version of the client
• ClientCipher - Cipher suite used by the client

Acquiring Cloudflare Logs

You can configure Cloudflare to forward logs using several different methods. Splunk HEC (HTTP Event Collector) is an easy one to set up both on Cloudflare and Grepr. You create a HEC endpoint in Grepr by adding a Splunk integration and then configuring the Cloudflare log push to send to the Grepr endpoint. If you want them to end up in Splunk ultimately, the integration has already been created. If you want to send the logs somewhere else, like Datadog or New Relic, create an integration for that platform. Finally, create a Grepr pipeline to process the logs. Use the Splunk HEC as the source and whichever ultimate destination as the sink. If it is not Splunk, remove Splunk from the sink. The logs will now start flowing into Grepr.

Using Cloudflare Logs To Manage Cost

Cloudflare logs provide great detail on the traffic flowing through, which you can use for configuration modification changes to optimize costs. Reducing unnecessary bandwidth, compute, and log storage. To find the signal in the noise, consider the following fields and how they can help inform decision-making.

Typically, your cloud platform will be charging for egress, so the more data you send, the more it’s going to cost. Cross-referencing EdgeResponseBytes, which is the size of the payload sent to Cloudflare from the origin server, with OriginIP will inform you which services are utilizing the egress the most. Your engineers might be able to use this information to make some optimizations. Another source of high egress is cache misses (CacheStatus: MISS), where the payload was served by the origin and not from the Cloudflare cache. Check that Cache Everything is enabled for static assets, such as CSS, JavaScript, and images. Additionally, check the setting of Cache-Control headers. For example, setting this to max-age=31536000 would cache assets for a year.

Using Grepr To Stop Escalating Log Costs

Sending large amounts of logs from Cloudflare to your log aggregation and storage platform (Splunk, Datadog, New Relic) can seriously impact your observability costs. However, Grepr reduces log volume by 90% while retaining 100% insight. Verbose messages are sent through as summaries, while unique messages are passed through, allowing you to collect and extract intelligence from different log sources without being overly concerned with the cost implications.