Your Logfiles May Be Missing Important Data
http://www.isedb.com/db/articles/1762/1/Your-Logfiles-May-Be-Missing-Important-Data/Page1.html
Ross Dunn
Ross Dunn is the CEO of StepForth Placement Inc., a search engine marketing company founded in 1997 and based in Victoria, British Columbia, Canada. StepForth provides cutting-edge search engine optimization services that provide highly successful, targeted results for its clientele. Ross Dunn is a Certified Internet Marketing and Business Strategist (CIMBS) with a background in web design and online marketing. His broad Internet experience in combination with a talented staff has made StepForth a name synonymous with top results. By Ross Dunn
Published on 12/27/2007
In this article I am going to explain logfiles and their importance in website analytics from my perspective as a ClickTracks user. ClickTracks can do a lot to determine what is or is not
working on a website; much more than expected in most cases. The one
thing, however, that ClickTracks or any other logfile-based analytics
tool cannot do is interpret information in your logfiles if it is not
recorded. Unfortunately this is a common occurrence and many site
owners have no idea that their hosting company is not saving
information that could help them now or later when they find they need
it.
Your Logfiles May Be Missing Important Data
In this article I am going to explain logfiles and their importance in website analytics from my perspective as a ClickTracks user. Before I begin, however, I want you to know that although I offer essential analytic consulting, I am a certified ClickTracks Analytics Professional
and have dabbled in books on analytics, I don't consider myself to be
an analytics expert. In fact, I constantly find myself humbled by how
much more there is to know. That said, I do know more than the average
site owner and I hope that this article can shine a little light on
this often confusing subject and save you some future headaches.
As many of you may know I am a huge fan of the logfile version of ClickTracks Professional, a website analytics package that I find indispensable for myself and my clientele. ClickTracks can do a lot to determine what is or is not working on a website; much more than expected in most cases. The one thing, however, that ClickTracks or any other logfile-based analytics tool cannot do is interpret information in your logfiles if it is not recorded. Unfortunately this is a common occurrence and many site owners have no idea that their hosting company is not saving information that could help them now or later when they find they need it.
The reality is that over ninety percent of the hosting companies I have dealt with have not been saving the vital data that a higher level analytics program needs; to work at peak performance if at all. In this posting I will provide an overview on this issue so you have enough information to approach your hosting company about making the required updates to their systems. If you are unsure you can even refer them to this article and/or the set of questions and details I provided below.
What is a Logfile?
First, let me explain the very basic idea of what a logfile is and how one is created.
Whenever you visit a website your browser requests information from the server hosting the website. This request is passed onto the server and in turn the server delivers the information requested by your browser. Whenever this exchange takes place your server saves the request along with a host of information about the requesting browser such as:
How Can Logfiles Help Improve Your Website?
Now that you know what data is collected it is time to explain, in general terms, how this data can be used to help your website. There is a wide variety of information that can be gleaned from a complete logfile such as:
Many hosting company’s are smart enough to include a basic web analytics program with every account. These programs are decent for anyone who wants to simply find out the traffic to their website and a myriad of other basic stats. However, there is often a pitfall to these basic programs. You see in order to save on computer performance the hosting company usually sets their servers to collect only the minimal data these basic systems require. As a result, more complex logfile-based analytics programs may find themselves starved of the data they need to operate fully. This is where my clients have found themselves before; they have sub-par logfiles and are forced to try and convince their hosting company to change their data collection methods to meet more advanced standards.
If you have no interest in website analytics you may find this whole scenario to be a non-issue. I completely understand, however, put yourself a year or even a month down the road when your website is taking off and you need to know more about the visitors to your website. You just might find yourself in this same frustrating scenario and it will seem absolutely insane how hard you have to push to get this data properly collected. Unfortunately, unless you are leasing your own private (dedicated) server from the hosting company they tend to set up their shared servers with only the basic needs of the majority in mind. As a result, the only way to force change is if more customers consider it a basic need – thus the reason for this article. Help me affect change so that you save yourself a headache in the future!
How to Be Sure Your Server is Collecting the Right Information
Most of you cannot check your logfiles for completeness with an analytics program so you will have to trust your server administrators to do their due diligence based on the following question.
Note: If you like you can just copy and paste the following question (noted in red) and send it to your hosting company support staff:
Hello,
I would like to make sure my website’s logfiles have the necessary information to run a higher end web analytics program. Is your server set up to collect the data on my website? I need this data to properly analyze the traffic on my website.
Sincerely,
<>
My Hosting Company Disregarded This as Nonsense
I fully expect some will and that is because many website owners still care little or nothing about web site statistics so they have not even used the basic data to its fullest yet - and hosting company's are aware of this. In fact, a good friend who owns a hosting company himself guessed around 95% of his website clients never even look at their stats. This is all true, however, does that mean that important data should not be collected for those who do want to delve deeper into analytics? I don't believe so and the changes you are requesting will only increase the size of the logfiles for your website a small amount. Unless of course you don't even have logfiles which is enough for me to recommend you take your services elsewhere.
Why Not Use Google Analytics Instead?
Google Analytics is an awesome solution for many small businesses. It does not require logfiles and it takes a marginal amount of work to begin acquiring proper data. In fact, I think it is a great tool for the majority of businesses that want to wade into a mid range analytics solution providing you are comfortable with Google having access to your stats. That said, there is one MAJOR flaw in using Google Analytics... it does not have reliable click fraud reporting. You see many of my clients use ClickTracks to monitor their pay per click campaign for click fraud which is not something I would ever trust Google to police itself on. That does not mean I do not use Google Analytics. In fact, whenever possible I use both ClickTracks and Google Analytics in tandem for redundancy especially when certain capabilities such as cookie tracking are not available from a hosting provider - Google includes cookies by default.
In Summary
Many website owners have no idea what they will or will not need in the future to properly administrate their online marketing campaigns. This article discusses a simple adjustment to the accumulation of website logfiles that I strongly believe all competent hosting companies should implement in order to provide scalability for their clientele. The adjustment will provide the additional information that a competent analytics solution will need to provide accurate statistics.
As many of you may know I am a huge fan of the logfile version of ClickTracks Professional, a website analytics package that I find indispensable for myself and my clientele. ClickTracks can do a lot to determine what is or is not working on a website; much more than expected in most cases. The one thing, however, that ClickTracks or any other logfile-based analytics tool cannot do is interpret information in your logfiles if it is not recorded. Unfortunately this is a common occurrence and many site owners have no idea that their hosting company is not saving information that could help them now or later when they find they need it.
The reality is that over ninety percent of the hosting companies I have dealt with have not been saving the vital data that a higher level analytics program needs; to work at peak performance if at all. In this posting I will provide an overview on this issue so you have enough information to approach your hosting company about making the required updates to their systems. If you are unsure you can even refer them to this article and/or the set of questions and details I provided below.
What is a Logfile?
First, let me explain the very basic idea of what a logfile is and how one is created.
Whenever you visit a website your browser requests information from the server hosting the website. This request is passed onto the server and in turn the server delivers the information requested by your browser. Whenever this exchange takes place your server saves the request along with a host of information about the requesting browser such as:
- the internet address (loosely connected to location) known as the IP
- browser type (Internet Explorer or Firefox or..)
- the screen resolution of the browser used
- time and date of the request
- the page requested for viewing
- the website the visitor came from (known as a Referrer)
- if applicable, the keyword(s) that were used to find your website on a search engine
- etc.
How Can Logfiles Help Improve Your Website?
Now that you know what data is collected it is time to explain, in general terms, how this data can be used to help your website. There is a wide variety of information that can be gleaned from a complete logfile such as:
- How long visitors stay at your website or on a particular page.
- What pages they visited.
- Where visitors are viewing your website from geographically.
- What keywords were used to visit your website and which search engines were driving the highest volume and/or quality traffic.
- Which pages had the highest or least traffic.
- The average time a visitor stays at your website: often a great indication of the 'stickiness' of your website.
- You can determine the effectiveness of your pay-per-click campaign by tracking visitors specifically delivered from the campaign.
- Identify potential pay-per-click fraud using tools like ClickTracks Professional that has a click fraud reporting tool.
- and much more...
Many hosting company’s are smart enough to include a basic web analytics program with every account. These programs are decent for anyone who wants to simply find out the traffic to their website and a myriad of other basic stats. However, there is often a pitfall to these basic programs. You see in order to save on computer performance the hosting company usually sets their servers to collect only the minimal data these basic systems require. As a result, more complex logfile-based analytics programs may find themselves starved of the data they need to operate fully. This is where my clients have found themselves before; they have sub-par logfiles and are forced to try and convince their hosting company to change their data collection methods to meet more advanced standards.
If you have no interest in website analytics you may find this whole scenario to be a non-issue. I completely understand, however, put yourself a year or even a month down the road when your website is taking off and you need to know more about the visitors to your website. You just might find yourself in this same frustrating scenario and it will seem absolutely insane how hard you have to push to get this data properly collected. Unfortunately, unless you are leasing your own private (dedicated) server from the hosting company they tend to set up their shared servers with only the basic needs of the majority in mind. As a result, the only way to force change is if more customers consider it a basic need – thus the reason for this article. Help me affect change so that you save yourself a headache in the future!
How to Be Sure Your Server is Collecting the Right Information
Most of you cannot check your logfiles for completeness with an analytics program so you will have to trust your server administrators to do their due diligence based on the following question.
Note: If you like you can just copy and paste the following question (noted in red) and send it to your hosting company support staff:
Hello,
I would like to make sure my website’s logfiles have the necessary information to run a higher end web analytics program. Is your server set up to collect the data on my website? I need this data to properly analyze the traffic on my website.
- Date and Time
- Client IP Address
- HTTP Method
- Requested file and Query string
- User Agent
- Referrer
- Status code
- Cookie (preferable, but not required)
Sincerely,
<>
My Hosting Company Disregarded This as Nonsense
I fully expect some will and that is because many website owners still care little or nothing about web site statistics so they have not even used the basic data to its fullest yet - and hosting company's are aware of this. In fact, a good friend who owns a hosting company himself guessed around 95% of his website clients never even look at their stats. This is all true, however, does that mean that important data should not be collected for those who do want to delve deeper into analytics? I don't believe so and the changes you are requesting will only increase the size of the logfiles for your website a small amount. Unless of course you don't even have logfiles which is enough for me to recommend you take your services elsewhere.
Why Not Use Google Analytics Instead?
Google Analytics is an awesome solution for many small businesses. It does not require logfiles and it takes a marginal amount of work to begin acquiring proper data. In fact, I think it is a great tool for the majority of businesses that want to wade into a mid range analytics solution providing you are comfortable with Google having access to your stats. That said, there is one MAJOR flaw in using Google Analytics... it does not have reliable click fraud reporting. You see many of my clients use ClickTracks to monitor their pay per click campaign for click fraud which is not something I would ever trust Google to police itself on. That does not mean I do not use Google Analytics. In fact, whenever possible I use both ClickTracks and Google Analytics in tandem for redundancy especially when certain capabilities such as cookie tracking are not available from a hosting provider - Google includes cookies by default.
In Summary
Many website owners have no idea what they will or will not need in the future to properly administrate their online marketing campaigns. This article discusses a simple adjustment to the accumulation of website logfiles that I strongly believe all competent hosting companies should implement in order to provide scalability for their clientele. The adjustment will provide the additional information that a competent analytics solution will need to provide accurate statistics.