Download raw server access logs

Hello there!

Thank you for offering this amazing service. Although current implementation of Analytics marginally covers my use cases, I felt obliged to subscribe for it in order to give something back and keep Netlify running. :wink: However, I do have in mind one suggestion that would make Analytics way more useful.

I think it would be very helpful for everyone if you would offer raw access logs export. This would provide the information required for any exotic use case and will temporarily cover all use cases until items in roadmap get implemented. Do you want to compare custom date ranges or get stats for individual articles? Download the raw server logs and use AWStats or GoAccess or Scratchy or your favorite logs analyzer or even your own script!

Various download options may be implemented, like for example one file per month, or last month plus month-to-date, or only year-to-date, or only rolling 30-days… Any option would work if someone would regularly download the generated files. You are in a better position to judge the effort required and other variables for each option and to choose what to implement. Now, if you are thinking to support logs exporting with custom date ranges, or with API endpoints, that would fill me with great joy. :smiley:

Offering raw logs exports for Analytics customers is not competitive to the Analytics product, rather a supplement to it. Logs exports offer the option to do further analysis than what provided by the predetermined charts. My opinion is that log exports will make the Analytics product more lucrative. Imagine the next time you’ll answer a support request instead of “We can’t do this right now…” with “We can’t do this right now, but you can download…”; which sounds more appealing?

I know that you’re already gathering the data in order to analyze them and create the current reporting. Isn’t this feature a low hanging fruit that would worth prioritizing before implementing the rest of the Analytics roadmap?

Thank you for hearing this and again thank you for the great service you offer.

3 Likes

Howdy & thanks so much for the thoughtful feedback!

We have put a lot of consideration into offering raw log download, and it is not practical for us at present. As a start, we are not permitted to give you any personally identifying information about your visitors, such as IP address and in some cases user agent due to the GDPR and other privacy laws, so the logs would need to be scrubbed first, at a minimum.

I have added your voice to the open feature request asking for this from some other customers, but I do not think it is likely to happen soon. Your best bet for getting better analytics information is asking for analytics to be improved, rather than asking for raw access logs - since we are still actively working on the analytics product.

Any feedback on that topic (improving analytics without offering raw access logs) would be most welcome and I would personally review it with the analytics development and project management team, so you can be sure it would be heard if you wanted to provide it :slight_smile:

1 Like

@fool it kinda feels like an excuse not to give them and enforce users to register for analytics, and I’m all for GDPR. It’s like AWS saying we can host your S3 buckets but we can’t give you access to the logs. If I host my website at Netlify, I should be able to get the logs.

Right now, not having raw server access logs means I either have to add a JS version like Google Analytics and add trackers on my website, or pay Netlify (which I don’t mind) specifically for that feature and have a very limited version of analytics.

Love Netlify and the service you guys built, but this is the kind of move which makes me think maybe I should use Gitlab/Github pipelines and just upload my static website to an S3/Cloudfrout bucket, get access logs and use Awstats.

1 Like

Hi Fabien,

I understand your feelings, but this was our policy for years before we had an analytics product too, so your accusation while reasonable to guess at, is not true.

Since my post almost a year ago, log export is closer to being implemented - we are talking about how to do it scalably and sustainably, taking into account privacy laws. I don’t have any proposed timeline for completion, nor can I guarantee it will be available at all account levels, but your use case is not weird and we do want to solve it, with or without our analytics product. It is, however, a very hard problem (we have terabytes of access logs PER DAY), so there is no quick solution.

Of course you need to make the decision that is correct for your business. If building your own Netlify (there’s a heck of a lot more to it than S3 + Pipelines if you want to do it well) is the only path you see, I will advise you to follow that path, because your business’ needs are real and immediate, regardless of what service it is possible, convenient, or planned, for us to provide.

Hope you can stay with us, but please do NOT take this as a promise that this problem will be solved for you in the way you want on any timeline, so you need to make the correct choice for yourself.

1 Like

Thanks @fool for the clear answer. Hope to see it implemented one day.

Hello!

It would be awesome if you can provide server logs, because your Analytics it’s nice but not sufficient from an SEO perspective: how do we track Googlebot hits (something basic), for instance? There are many insights we can get through server logs that we are now missing… Hope you can make some kind of move regarding this matter.

Best,
J

3 Likes

Heya, +1 for tracking Googlebot hits please :smiley:

2 Likes

@javier ist right. I also came here because of this missing feature with regards to GoogleBot Crawler. There are tools out there which you feed the access.log and it will give you GoogleBot Crawler Stats. Very much needed from SEO perspective. Not having this is a real shortcomming and GDPR cant be an excuse. Anonymizing IPs before making them available shouldnt be too hard right?

2 Likes

Hi, @logemann. I’ve added this +1 to the feature request.

2 Likes

Hi fool, +1 to the feature request.
Im also here for the raw logs export.

1 Like

Hi, @GuestX. I’ve added your +1 to the list and we will follow-up in this topic if or when this becomes available.

Hi fool, +1 to the feature request.
Im also here for the raw logs export.

1 Like

Hi, I also am interested in something closer to the raw logs.

My particular use cases:

  1. I have a few services I use for testing uptime and page load speeds, and I currently have no way to exclude them when looking at traffic. Likewise for excluding crawlers.

  2. I have a product I sell exclusively in the US. Seeing all traffic in the US lumped together is effectively useless for me in terms of marketing efforts. Based on my limited understanding of the California privacy laws (and general “personal information” laws), reverse geocoding IPs is not considered personal information so long as it’s coarse enough data. When I was an iPhone developer, we provided location data limited only to metro area. Being able to see traffic by metro area, or at least by state, would be immensely helpful.

As a matter of principle, I would like to limit use of granular tracking services and tracking cookies. I would like to see Netlify’s offerings meet those needs.

Thanks for your consideration.

Nearly 2 years on from the OP, it seems that netlify isn’t really going in any direction. Doesn’t seem to improve at all with any of the featuresd requested.

Hi @namelessshameless,

We do have Log Drains at the moment that might be what you’re asking for: Log Drains | Netlify Docs.