Laravel Bot Analytics Without Storing Raw User-Agent Strings

After ICanUp analytics gained a Human versus Bot split, the next question was obvious: which bots are actually generating that traffic?

A single Bot metric was no longer detailed enough. Search crawlers matter for SEO, link preview services are useful to distinguish, and the rise of AI crawlers made their share interesting as well.

The simplest approach would be to store the full User-Agent for every view and analyze it later.

I deliberately chose a different design.

When an HTTP request reaches Laravel, User-Agent carries technical information about the client.

It can help derive:

• device type;
• browser family;
• operating-system family;
• whether the request looks automated;
• whether the bot matches a known crawler family.

The raw value is genuinely useful for that step.

But it is useful while processing the request. That does not imply it needs to become a persisted analytics dimension.

This became a core rule in my analytics design: raw request data may be necessary to derive a signal, but historical storage should contain only information that is actually useful for reporting.

The canonical top-level distinction remains is_bot.

This is enough when I want the overall Human versus Bot distribution.

For bot analysis itself, however, a boolean is not detailed enough.

The second classification level is useful only for requests already identified as Bot.

Conceptually, normalized results may look like this:

I do not need a synthetic human value inside crawler_family.

is_bot already carries that distinction. The second dimension exists only to explain Bot traffic.

A full User-Agent is tempting because it preserves maximum information for possible future analysis.

For ordinary content analytics, however, it creates several problems.

My analytics stores aggregates by date and dimensions.

Every new unique dimension value can create another aggregate combination.

Persisting full User-Agent strings would fragment storage by details I do not even intend to expose in the primary analytics interface.

An important case is an automated client whose signature is not part of the supported list.

I do not want to extract a name from its User-Agent and create a new analytics value.

This keeps even unusual or deliberately crafted headers from expanding the analytics vocabulary with arbitrary values.

There is another important boundary.

User-Agent is supplied by the client and can be spoofed.

A request calling itself Googlebot is not automatically verified as Googlebot.

That classification is still useful for routine analytics.

But investigating suspicious activity requires other evidence and a separate verification process.

For a monthly traffic picture, normalized aggregates are enough.

When thousands of suspicious requests suddenly arrive, the technical details of individual HTTP requests become important.

The bot approach is not unique.

The same request header can produce bounded values for other reports:

To answer "how many views came from Chrome?", I do not necessarily need the exact build version of every browser unless that detail has a separate analytical purpose.

I apply the same minimization principle to geography.

A normalized two-letter country code from a trusted infrastructure source is enough for country reports.

Before adding another field to analytics, I try to answer three questions.

• Which concrete decision will this metric help me make?
• Can a normalized category replace the raw value?
• Will this dimension create uncontrolled aggregate cardinality?

If the first question has no answer, storing the value "just in case" is not enough justification for me.

I do not create a third top-level visitor class for AI crawlers beside Human and Bot.

At the canonical visitor-class level, it is automated traffic.

This keeps the top-level Human versus Bot view simple while allowing search crawlers, AI crawlers, and other families to be analyzed inside Bot traffic.

When I display crawler-family shares, the denominator needs to be Bot traffic itself.

Dividing 1000 by all 10000 views produces a different metric with a different meaning.

crawler_family is more useful than one global list.

I want it to work with the same period and content filters as the rest of analytics.

This does not require a separate bot datastore. The normalized family becomes another supported dimension in the shared analytics system.

I do not want one table for Googlebot, another for AI crawlers, and a third for other bots.

Classification belongs in the shared analytics enrichment stage before persistence.

Analytics is supporting functionality for the public site.

If a strange or malformed User-Agent cannot be classified, the visitor should still receive the page.

A deterministic fallback or a missing optional dimension is preferable to turning a classifier failure into an HTTP 500 response.

The supported-family list should not grow through unrelated checks scattered across controllers.

Conceptually, I want one boundary that turns a raw signal into a bounded result.

In the second form, the application defines the available values rather than the external client.

A functional test proving that Googlebot reaches the correct group is not enough.

I also want to verify that the raw value never leaks into historical analytics.

A normalized other_bot value protects storage from an unlimited vocabulary.

The trade-off is obvious: if that group suddenly generates most bot traffic, the aggregate report does not identify the exact source.

I still do not want to solve that by permanently storing every raw request header.

When aggregated analytics shows an anomaly such as a sudden other_bot spike, that becomes a signal for operational investigation.

Server logs or another controlled operational source can then be inspected for the relevant time window.

If a new crawler is stable, analytically useful, and has a reliable signature, it can be added to the controlled classifier.

If it was a one-off scanner, long-term analytics does not need a permanent new category because of one incident.

I do not need to choose between knowing nothing about traffic and retaining every technical detail of every request.

There is a practical middle path:

• receive the raw signal with the request;
• turn it into the required classification;
• persist only the normalized result;
• aggregate it for reporting;
• leave detailed technical investigations to the operational layer.

• The complete raw User-Agent of every view.
• Raw IP as an analytics dimension.
• Arbitrary crawler names parsed from headers.
• Full referrer URLs with arbitrary query parameters.
• A persistent profile for unauthenticated visitors.
• Technical values merely because the browser or HTTP request makes them available.

• Define the concrete product or technical question a dimension should answer.
• Use raw User-Agent only as an input signal for classification.
• Keep Human/Bot as the canonical top-level split.
• Detail only Bot traffic through a bounded crawler_family.
• Keep crawler_family null for Human traffic.
• Never create a new dimension value from arbitrary User-Agent text.
• Map unknown bots to a stable fallback.
• Do not treat User-Agent as verified crawler identity.
• Keep long-term analytics separate from operational logs.
• Do not store raw IP merely for traffic grouping.
• Test both classification results and the absence of raw values in storage.
• Control the cardinality of every new dimension.

In ICanUp, I want to understand how much traffic comes from humans, how much comes from bots, and which known crawler families visit public content.

That does not require storing the full User-Agent of every view.

The raw header is used as a transient input for classification. Historical analytics then keeps bounded values such as is_bot, crawler_family, device type, browser family, and operating-system family.

An unknown bot does not create an arbitrary new category. It maps to a predictable fallback. Human traffic does not receive a crawler family at all.

When a real investigation into a scanner or suspicious request burst is required, that belongs to operational logs rather than long-term content aggregates.

My main principle is simple: store not the maximum amount of data available, but the minimum amount that is sufficient to answer a useful question.