1 rizwank 1.1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
2 <html>
3 <head>
4 <meta name="description" content="AWStats Documentation - FAQ page">
5 <meta name="keywords" content="awstats, awstat, faq, error, errors, frequently, asked, questions, support, help, problem, solution, troubleshooting, rotate log">
6 <meta name="robots" content="index,follow">
7 <meta name="title" content="AWStats Documentation - FAQs">
8 <title>AWStats Documentation - FAQs</title>
9 <link rel="stylesheet" href="styles.css" type="text/css">
10 <!-- $Revision: 1.155 $ - $Author: eldy $ - $Date: 2005/01/15 23:30:52 $ -->
11 </head>
12
13 <body topmargin=10 leftmargin=5>
14
15
16 <table style="font: 10pt arial,helvetica,verdana" cellpadding=0 cellspacing=0 border=0 bgcolor=#FFFFFF width=100%>
17
18 <!-- Large -->
19 <tr style="font: 10pt arial,helvetica,verdana">
20 <td bgcolor=#9999cc align=center><a href="/"><img src="images/awstats_logo4.png" border=0></a></td>
21 <td bgcolor=#9999cc align=center>
22 rizwank 1.1 <br>
23 <font style="font: 16pt arial,helvetica,sans-serif" color=#EEEEFF><b>AWStats logfile analyzer 6.3 Documentation</b></font><br>
24 <br>
25 </td>
26 <td bgcolor=#9999cc align=center>
27
28 </td>
29 </tr>
30
31 </table>
32
33
34 <br><br><H1 style="font: 26px arial,helvetica,sans-serif">Frequently Asked Questions + Troubleshooting</H1>
35
36
37 <br>
38 <u>ABOUT QUESTIONS:</u><br>
39 <ul>
40 FAQ-ABO100 <a href="#SERVERSOS">Which server log files or operating systems are supported ?</a><br>
41 FAQ-ABO150 <a href="#LOGFORMAT">Which log format can AWStats analyze ?</a><br>
42 FAQ-ABO200 <a href="#LANG">Which languages are available ? How to add my own language ?</a><br>
43 rizwank 1.1 FAQ-ABO250 <a href="#PHPNUKE">Can AWStats be integrated with PHP Nuke ?</a><br>
44 </ul>
45 <br>
46
47 <u>COMMON SETUP/USAGE QUESTIONS:</u><br>
48 Here, you can find the most common questions and answers about AWStats setup/usage process.<br>
49 <ul>
50 FAQ-COM025 <a href="#NOLOG">How to use AWStats with no server log</a><br>
51 FAQ-COM050 <a href="#LIMITLOG">What is the log size limit AWStats can analyze ?</a><br>
52 FAQ-COM090 <a href="#FTP">Setup for FTP server log files (proftpd, vsftpd, ...).</a><br>
53 FAQ-COM100 <a href="#MAIL">Setup for MAIL log files (Postfix, Sendmail, QMail, MDaemon, Exchange).</a><br>
54 FAQ-COM110 <a href="#MEDIASERVER">Setup for MEDIA SERVER log files (Realmedia, Windows media, Darwin streaming server).</a><br>
55 FAQ-COM120 <a href="#ROTATE">How to rotate my logs without loosing data.</a><br>
56 FAQ-COM130 <a href="#CRONTAB">How to run AWStats frequently ?</a><br>
57 FAQ-COM140 <a href="#EXCLUDEHOSTS">How to exclude my IP address (or whole subnet mask) from stats ?</a><br>
58 FAQ-COM145 <a href="#EXTRA">How to use the Extra Sections features ?</a><br>
59 FAQ-COM150 <a href="#BENCHMARK">Benchmark question.</a><br>
60 FAQ-COM200 <a href="#DNS">How reverse DNS Lookup works, unresolved IP Addresses ?</a><br>
61 FAQ-COM250 <a href="#DIFFERENT_RESULTS">Different results than other log analyzers (Analog, Webalizer, WUsage, wwwStats...).</a><br>
62 FAQ-COM300 <a href="#DIFFERENCE_HOURS">Difference between local hour and AWStats reported hour.</a><br>
63 FAQ-COM350 <a href="#OLDLOG">How can I process old log file ?</a><br>
64 rizwank 1.1 FAQ-COM360 <a href="#MULTILOG">How can I process several log files in one run ?</a><br>
65 FAQ-COM400 <a href="#LOADLOG">How can I update my statistics when I use a load balancing system that splits my logs ?</a><br>
66 FAQ-COM500 <a href="#RESET">How can I reset all my statistics ?</a><br>
67 FAQ-COM600 <a href="#DAILY">How can I compile and build statistics on a daily basis only ?</a><br>
68 FAQ-COM700 <a href="#EDITHISTORY">Can I safely remove a line in AWStats history files (awstatsMMYYYY*.txt) ?</a><br>
69 </ul>
70 <br>
71
72 <u>ERRORS/TROUBLESHOOTING QUESTIONS:</u><br>
73 Here, you can find the most common questions and answers about errors or problems using AWStats.<br>
74 <ul>
75 FAQ-SET050 <a href="#MISSINGDOLLAR">Error "Missing $ on loop variable ..."</a><br>
76 FAQ-SET100 <a href="#CGISOURCE">I see Perl script's source instead of its execution in my browser.</a><br>
77 FAQ-SET150 <a href="#SPAWNERROR">Error "...couldn't create/spawn child process..." with Apache for windows.</a><br>
78 FAQ-SET200 <a href="#INTERNAL">"Internal Error" or "Error 500" in a browser connecting to Apache.</a><br>
79 FAQ-SET210 <a href="#SPEED">"Internal Error" after a long time in my browser (See FAQ-COM100 "AWStats speed/timeout problems").</a><br>
80 FAQ-SET220 <a href="#CRASH">Crash while running awstats.pl or page content only partialy loaded</a><br>
81 FAQ-SET250 <a href="#PERSONALIZEDLOG">Log format setup or errors.</a><br>
82 FAQ-SET270 <a href="#CORRUPTEDDROPPED">Only corrupted/dropped records</a><br>
83 FAQ-SET280 <a href="#NOTSAMENUMBER">Error "Not same number of records of...".</a><br>
84 FAQ-SET300 <a href="#COULDNOTOPEN">Error "Couldn't open file ..."</a><br>
85 rizwank 1.1 FAQ-SET320 <a href="#MALFORMEDUTF8">Error "Malformed UTF-8 character (unexpected..."</a><br>
86 FAQ-SET350 <a href="#EMPTY_STATS">Empty or null statistics reported.</a><br>
87 FAQ-SET360 <a href="#PARTIAL_STATS">Statistics reported except for os, browsers, robots and keywords/keyphrases.</a><br>
88 FAQ-SET400 <a href="#REDIRECT">Pipe redirection to a file give me an empty file.</a><br>
89 FAQ-SET450 <a href="#NO_ICON">No pictures/graphics shown.</a><br>
90 FAQ-SET700 <a href="#MIGRATEDOUBLED">My visits are doubled for old month I migrated from 3.2 to 5.x</a><br>
91 FAQ-SET750 <a href="#OUTOFMEMORYCYGWIN">AWStats run out of memory during update process with cygwin Perl.</a><br>
92 FAQ-SET800 <a href="#SPEED">AWStats speed/timeout problems.</a><br>
93 </ul>
94 <br>
95
96 <u>SECURITY QUESTIONS:</u><br>
97 Here, you can find the common questions about security problems when setting or using AWStats.<br>
98 <ul>
99 FAQ-SEC100 <a href="#CSSATTACK">Can AWStats be used to make Cross Site Scripting Attacks ?</a><br>
100 FAQ-SEC150 <a href="#SECUSER">How can I prevent some users to see statistics of other users ?</a><br>
101 FAQ-SEC200 <a href="#WORMS">How to manage log files (and statistics) corrupted by worms attacks like 'Code Red Virus like'.</a><br>
102 </ul>
103 <br>
104
105 <hr>
106 rizwank 1.1 <br><br>
107
108
109 <a name="SERVERSOS"></a><br>
110 <b><u>FAQ-ABO100 : WHICH SERVER LOG FILES OR OS ARE SUPPORTED ?</u></b><br>
111 AWStats can works with :<br>
112 <li> All web server able to write log file with a <u>combined log format (XLF/ELF)</u> like Apache,
113 a <u>common log format (CLF)</u> like Apache or Squid, a <u>W3C log format</u> like IIS 5.0 or higher,
114 or any other log format that contains all information AWStats expect to find.<br>
115 <li> Most of all others Web/Wap/Proxy/Streaming servers.<br>
116 <li> Some FTP, Syslog or Mail log files.<br>
117 Because AWStats is in Perl, it can works on all Operating Systems.<br>
118 Examples of used platforms (bold means 'tested by author', others were reported by AWStats users to work correctly) :<br>
119 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
120 <u>OS:</u><br>
121 <b>Windows 2000</b>, <b>Windows NT 4.0</b>, Windows Me, <b>Linux (RedHat, Mandrake, Debian, Suse...)</b>, Macintosh, <b>Solaris</b>, <b>Aix</b>, BeOS, FreeBSD, ...<br>
122 <u>Web/Wap/Proxy/Streaming servers</u><br>
123 <b>Apache 1.3.x and 2.x</b>, <b>IIS 5.0 and 6.0</b>, WebStar, WebLogic, WebSite, <b>Windows Media Server</b>, Tomcat, <b>Squid</b>,
124 Sambar, Roxen, Resin, RealMedia server, Oracle9iAS, <b>Lotus Notes/Domino</b>, Darwin, IPlanet, IceCast, ZeroBrand, Zeus, Zope, Abyss...<br>
125 <u>FTP servers</u><br>
126 <b>ProFTPd</b>, vsFTPd...<br>
127 rizwank 1.1 <u>Mails servers</u><br>
128 <b>Postfix</b>, <b>Sendmail</b>, QMail, <b>Mdaemon</b>, www4mail, ...<br>
129 <u>Perl interpreters (all Perl >= 5.005):</u><br>
130 <b>ActivePerl 5.6</b>, <b>ActivePerl 5.8</b>, <b>Perl 5.8</b>, <b>Perl 5.6</b>, <b>Perl 5.005</b>, <b>mod_perl</b> and mod_perl2 for Apache, ...<br>
131 </td></tr></table>
132 <br>
133
134 <a name="LOGFORMAT"></a><br>
135 <b><u>FAQ-ABO150 : WHICH LOG FORMAT CAN AWSTATS ANALYZE ?</u></b><br>
136 AWStats setup knows predefined log format you can use to make AWStats config easier. However,
137 you can define your own log format, that's the reason why AWStats can analyze nearly all web, wap
138 and proxy server log files. Some FTP servers log files, Syslog or mail logs can also be analyzed.<br>
139 The only requirement is "Your log file must contain required information".<br>
140 This is very short examples of possible log format:<br>
141 <br>
142 <i>
143 Apache common log format (see Note*),<br>
144 Apache combined log format (known as NCSA combined log format or XLF or ELF format),<br>
145 Any other personalized Apache log format,<br>
146 Any IIS log format (known as W3C format),<br>
147 Webstar native log format,<br>
148 rizwank 1.1 Realmedia server, Windows Media Server, Darwin streaming server,<br>
149 ProFTPd server, vsFTPd server,<br>
150 Postfix, Sendmail, QMail, Mdaemon<br>
151 A lot of web/wap/proxy/streaming servers log format<br>
152 <br>
153 Note*: Apache common log format (AWStats can now analyze such log files but such log files does not
154 contain all information AWStats is looking for. The problem is in the content, not in the
155 format). I think analyzing common log files is not interesting because there is a lot of
156 missing information: no way to filter robots, find search engines, keywords, os, browser.
157 But a lot of users asked me for it, so AWStats support it.
158 However, a lot of interesting advanced features can't work: browsers, os's, keywords, robot detection...).
159 </i><br>
160 See also <a href="#PERSONALIZEDLOG">F.A.Q.: LOG FORMAT SETUP OR ERRORS </a>.<br>
161 <br>
162
163 <a name="LANG"></a><br>
164 <b><u>FAQ-ABO200 : WHICH LANGUAGES ARE AVAILABLE ?</u></b><br>
165 AWStats can make reports in 40 languages. This is a list of all of them, for last version, in
166 alphabetical order (The code you can use for <a href="awstats_config.html#Lang">Lang</a>
167 and <a href="awstats_config.html#ShowFlagLinks">ShowFlagLinks</a> parameter are
168 the ISO-639-1 language codes):<br>
169 rizwank 1.1 <i>
170 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
171 Albanian=al, Bosnian=ba, Bulgarian=bg, Catalan=ca,
172 Chinese (Taiwan)=tw, Chinese (Simpliefied)=cn, Czech=cz, Danish=dk,
173 Dutch=nl, English=en, Estonian=et, Euskara=eu, Finnish=fi,
174 French=fr, Galician=gl, German=de, Greek=gr, Hebrew=he, Hungarian=hu,
175 Icelandic=is, Indonesian=id, Italian=it, Japanese=jp, Korean=kr,
176 Latvian=lv, Norwegian (Nynorsk)=nn, Norwegian (Bokmal)=nb, Polish=pl,
177 Portuguese=pt, Portuguese (Brazilian)=br, Romanian=ro, Russian=ru,
178 Serbian=sr, Slovak=sk, Solvenian=si, Spanish=es, Swedish=se, Thai=th,
179 Turkish=tr, Ukrainian=ua, Welsh=cy.
180 </td></tr></table>
181 </i>
182 However, AWStats documentation is only provided in English.<br>
183 But, you may find small documentation for other languages made by contributors on
184 <a href="http://awstats.sourceforge.net/awstats_contrib.html#DOC">Documentation Contrib page</a>.<br>
185 <br>
186 If your language is not in this list, you can translate it yourself. For this, find what is your
187 2 letter language code: <a href="http://www.oasis-open.org/cover/iso639a.html">here</a>.<br>
188 Once, you get it, for example "gl" for Galician, copy the file awstats-en.txt into awstats-<i>gl</i>.txt,
189 in langs directory and translate every sentences inside. You can do same for files inside
190 rizwank 1.1 tooltips_f, tooltips_m and tooltips_w sub-directories. Then send your translated file(s) to eldy@users.sourceforge.net.<br>
191 <br>
192
193 <a name="PHPNUKE"></a><br>
194 <b><u>FAQ-ABO250 : CAN AWSTATS BE INTEGRATED WITH PHP NUKE ?</u></b><br>
195 The only plugin I know to integrate AWStats inside PHPNuke is here: <a href="http://phpnuke.org/modules.php?name=News&file=article&sid=7041">PhpNuke addon for AWStats</a><br>
196 <br>
197
198
199 <hr><br>
200
201
202 <a name="NOLOG"></a><br>
203 <b><u>FAQ-COM025 : HOW TO USE AWSTATS WITH NO SERVER LOG FILE</u></b><br>
204 <font class=CProblem>PROBLEM:</font><br>
205 I want to have AWStats statistics but i have no access to my server log file.<br>
206 <font class=CSolution>SOLUTION:</font><br>
207 Because AWStats is a log analyzer, if you don't have any way to read your server log file,
208 you have nothing to analyze and you should not be able to use AWStats.
209 However, this is a trick that you can use to have a log file be build. You must add
210 a tag to call a CGI script like pslogger into each of your web pages. This will
211 rizwank 1.1 allow you to have an artificial log file that can be analyzed by AWStats.<br>
212 You can find a Perl version of CGI pslogger enhanced by AWStats author <a href="/files/pslogger.pl">here</a>
213 or a php version of CGI pslogger made by Florent CHANTRET <a href="/files/pslogger.phps">here</a>.
214 <br>
215
216 <a name="LIMITLOG"></a><br>
217 <b><u>FAQ-COM050 : WHAT IS THE LOG SIZE LIMIT AWSTATS CAN ANALYZE</u></b><br>
218 <font class=CProblem>PROBLEM:</font><br>
219 I know I must run AWStats update process frequently on new log files, this means thoose
220 files have a regular size, but for my first update, I want/need to run update process
221 on old log files that are very large. Is there a limit on log file size AWStats can analyze ?<br>
222 <font class=CSolution>SOLUTION:</font><br>
223 No. There is no limit in AWStats. This means you can use it on large log files (test were
224 made on 10GB log files).<br>
225 However your system (Operating System or Perl version) might have a limit. For example, you can
226 experience size limit errors on files larger than 2 or 4 GB.
227 If limit is Perl only, try to use a Perl version compiled with "large file" option.<br>
228 If you can't find it nor build it, you can try to use a LogFile parameter that looks like this
229 <i>LogFile="cat /yourlogfilepath/yourlogfile |"</i> instead of
230 <i>LogFile="/yourlogfilepath/yourlogfile"</i><br>
231 <br>
232 rizwank 1.1
233 <a name="FTP"></a><br>
234 <b><u>FAQ-COM090 : SETUP FOR FTP SERVER LOG FILES (proftpd, vsftpd, ...)</u></b><br>
235 <font class=CProblem>PROBLEM:</font><br>
236 What do I have to do to use AWStats to analyze some FTP server log files ?<br>
237 <font class=CSolution>SOLUTION:</font><br>
238 AWStats can be used with some FTP server log files.<br>
239 <br>
240 <u>With ProFTPd</u>:<br>
241 <br>
242 1- Setup your server log file format:<br>
243 <br>
244 Modify the proftpd.conf file to add the following two lines :
245 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
246 <!--<br>With ProFTPd 1.2.2:-->
247 <i>LogFormat awstats "%t %h %u %m %f %s %b"</i> # WARNING: You must use a tab char between % tags and not a space !
248 <i><br>ExtendedLog /var/log/xferlog read,write awstats</i> # WARNING: ExtendedLog directive might need to be placed inside a virtual host context if you use them.
249 <!--<br>With ProFTPd 1.2.6:
250 <i><br>LogFormat awstats ""${%F %H-%M-%S}t %h %u %m %F %s %b"</i> # WARNING: You must use a tab char between % tags and not a space !
251 <i><br>ExtendedLog /var/log/xferlog read,write awstats</i> # WARNING: ExtendedLog directive might need to be placed inside a virtual host context if you use them.
252 -->
253 rizwank 1.1 </td></tr></table>
254 Then turn off old format Transfer log:
255 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
256 <i>TransferLog none</i> # WARNING: TransferLog directive might need to be placed inside a virtual host context if you use them.
257 </td></tr></table>
258 <br>
259 To have the change effective, stop your server, remove old log file /var/log/xferlog and restart the server.<br>
260 Download a file by FTP and check that your new log file looks like this:<br>
261 <i>[01/Jan/2001:21:49:57 +0200] ftp.server.com user RETR /home/fileiget.txt 226 1499</i><br>
262 <br>
263 2- Then setup AWStats to analyze the FTP log file:<br>
264 <br>
265 Copy config file "awstats.model.conf" to "awstats.ftp.conf".<br>
266 Modify this new config file:
267 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
268 <i>
269 LogFile="/var/log/xferlog"
270 <br>LogType=F
271 <br>LogFormat="%time1 %host %logname %method %url %code %bytesd"
272 <br>LogSeparator="\t"
273 <br>NotPageList=""
274 rizwank 1.1 <br>LevelForBrowsersDetection=0
275 <br>LevelForOSDetection=0
276 <br>LevelForRefererAnalyze=0
277 <br>LevelForRobotsDetection=0
278 <br>LevelForWormsDetection=0
279 <br>LevelForSearchEnginesDetection=0
280 <br>ShowLinksOnUrl=0
281 <br>ShowMenu=1
282 <br>ShowMonthStats=UVHB
283 <br>ShowDaysOfMonthStats=HB
284 <br>ShowDaysOfWeekStats=HB
285 <br>ShowHoursStats=HB
286 <br>ShowDomainsStats=HB
287 <br>ShowHostsStats=HBL
288 <br>ShowAuthenticatedUsers=HBL
289 <br>ShowRobotsStats=0
290 <br>ShowEMailSenders=0
291 <br>ShowEMailReceivers=0
292 <br>ShowSessionsStats=1
293 <br>ShowPagesStats=PBEX
294 <br>ShowFileTypesStats=HB
295 rizwank 1.1 <br>ShowFileSizesStats=0
296 <br>ShowBrowsersStats=0
297 <br>ShowOSStats=0
298 <br>ShowOriginStats=0
299 <br>ShowKeyphrasesStats=0
300 <br>ShowKeywordsStats=0
301 <br>ShowMiscStats=0
302 <br>ShowHTTPErrorsStats=0
303 <br>ShowSMTPErrorsStats=0
304 </i></td></tr></table>
305 <br>
306 Now you can use AWStats as usual (run the update process and read statistics).<br>
307 <br>
308 <br>
309 <u>With vsFTPd, or any FTP server that log with xferlog format</u>:<br>
310 <br>
311 1- Check your server log file format:<br>
312 <br>
313 Take a look at your FTP server log file. You must have a format that match the following example to
314 use this FAQ :<br>
315 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
316 rizwank 1.1 <i>Wed Jan 01 19:29:35 2001 1 192.168.1.1 102 /home/file1.txt b _ o r username ftp 0 * c</i>
317 </td></tr></table>
318 <br>
319 2- Then setup AWStats to analyze the FTP log file:<br>
320 <br>
321 If your FTP log file format looks good, copy config file "awstats.model.conf" to "awstats.ftp.conf".<br>
322 Modify this new config file:
323 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
324 <i>
325 LogFile="/var/log/xferlog"
326 <br>LogType=F
327 <br>LogFormat="%time3 %other %host %bytesd %url %other %other %method %other %logname %other %code %other %other"
328 <br>LogSeparator="\s"
329 <br>NotPageList=""
330 <br>LevelForBrowsersDetection=0
331 <br>LevelForOSDetection=0
332 <br>LevelForRefererAnalyze=0
333 <br>LevelForRobotsDetection=0
334 <br>LevelForWormsDetection=0
335 <br>LevelForSearchEnginesDetection=0
336 <br>ShowLinksOnUrl=0
337 rizwank 1.1 <br>ShowMenu=1
338 <br>ShowMonthStats=UVHB
339 <br>ShowDaysOfMonthStats=HB
340 <br>ShowDaysOfWeekStats=HB
341 <br>ShowHoursStats=HB
342 <br>ShowDomainsStats=HB
343 <br>ShowHostsStats=HBL
344 <br>ShowAuthenticatedUsers=HBL
345 <br>ShowRobotsStats=0
346 <br>ShowEMailSenders=0
347 <br>ShowEMailReceivers=0
348 <br>ShowSessionsStats=1
349 <br>ShowPagesStats=PBEX
350 <br>ShowFileTypesStats=HB
351 <br>ShowFileSizesStats=0
352 <br>ShowBrowsersStats=0
353 <br>ShowOSStats=0
354 <br>ShowOriginStats=0
355 <br>ShowKeyphrasesStats=0
356 <br>ShowKeywordsStats=0
357 <br>ShowMiscStats=0
358 rizwank 1.1 <br>ShowHTTPErrorsStats=0
359 <br>ShowSMTPErrorsStats=0
360 </i></td></tr></table>
361 <br>
362 Now you can use AWStats as usual (run the update process and read statistics).<br>
363 <br>
364
365
366 <a name="MAIL"></a><br>
367 <b><u>FAQ-COM100 : SETUP FOR MAIL LOG FILES (Postfix, Sendmail, Qmail, MDaemon, Exchange...)</u></b><br>
368 <font class=CProblem>PROBLEM:</font><br>
369 What do I have to do to use AWStats to analyze my mail log files ?<br>
370 <font class=CSolution>SOLUTION:</font><br>
371 <br>
372 This tip works with AWStats 5.5 or higher.<br>
373 <br>
374 <u>For Postfix, Sendmail, QMail or MDaemon log files</u><br>
375 <br>
376 You must setup AWStats to use a mail log file preprocessor (<i>maillogconvert.pl</i> is provided
377 into AWStats <i>tools</i> directory, but you can use the one of your choice):<br>
378 For this, copy config <i>"awstats.model.conf"</i> file to <i>"awstats.mail.conf"</i>.<br>
379 rizwank 1.1 Modify this new config file:
380 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
381 For standard Postfix, Sendmail, MDaemon and standard QMail logfiles, set<br>
382 <i>
383 LogFile="perl /path/to/maillogconvert.pl standard < /pathtomaillog/maillog |"<br>
384 </i>
385 If the logfiles are compressed, they can be processed this way<br>
386 <i>
387 LogFile="gzip -cd /var/log/maillog.0.gz | /path/to/maillogconvert.pl standard |"<br>
388 </i>
389 And for VAdmin QMail logfiles (multi-host/virtualhost mail servers running vadmin software), set<br>
390 <i>
391 LogFile="perl /path/to/maillogconvert.pl vadmin < /pathtomaillog/maillog |"<br>
392 </i>
393 Then, whatever is you mail server, you must also change:
394 <i>
395 <br>LogType=M
396 <br>LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd"
397 <br>LevelForBrowsersDetection=0
398 <br>LevelForOSDetection=0
399 <br>LevelForRefererAnalyze=0
400 rizwank 1.1 <br>LevelForRobotsDetection=0
401 <br>LevelForWormsDetection=0
402 <br>LevelForSearchEnginesDetection=0
403 <br>LevelForFileTypesDetection=0
404 <br>ShowMenu=1
405 <br>ShowMonthStats=HB
406 <br>ShowDaysOfMonthStats=HB
407 <br>ShowDaysOfWeekStats=HB
408 <br>ShowHoursStats=HB
409 <br>ShowDomainsStats=0
410 <br>ShowHostsStats=HBL
411 <br>ShowAuthenticatedUsers=0
412 <br>ShowRobotsStats=0
413 <br>ShowEMailSenders=HBML
414 <br>ShowEMailReceivers=HBML
415 <br>ShowSessionsStats=0
416 <br>ShowPagesStats=0
417 <br>ShowFileTypesStats=0
418 <br>ShowFileSizesStats=0
419 <br>ShowBrowsersStats=0
420 <br>ShowOSStats=0
421 rizwank 1.1 <br>ShowOriginStats=0
422 <br>ShowKeyphrasesStats=0
423 <br>ShowKeywordsStats=0
424 <br>ShowMiscStats=0
425 <br>ShowHTTPErrorsStats=0
426 <br>ShowSMTPErrorsStats=1
427 <br>
428 </i>
429 </td></tr></table>
430 Warning: For MDaemon mail server, you must use the new MDaemon log file that ends
431 with "-Statistics.log".<br>
432 <br>
433 Now you can use AWStats as usual (run the update process and read statistics).<br>
434 <br>
435
436 <u>For Exchange log files</u><br>
437 <br>
438 Despite the high number of possible log format provided with Exchange,
439 none of them is built enough seriously to offer an interseting analyze (missing informations,
440 messy data, no id to join multiple records for same mail, etc...).
441 For this reason, an "exact" log analysis is a joke with Exchange log files.
442 rizwank 1.1 However a little support is provided. In order to analyze Exchange traffic, you have to
443 enable "Message Tracking" (see article http://support.microsoft.com/default.aspx?scid=kb;EN-US;246856).<br>
444 Then copy config awstats.model.conf file to "awstats.mail.conf".<br>
445 Modify this new config file:
446 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
447 <i>
448 LogType=M
449 <br>LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd"
450 <br>LevelForBrowsersDetection=0
451 <br>LevelForOSDetection=0
452 <br>LevelForRefererAnalyze=0
453 <br>LevelForRobotsDetection=0
454 <br>LevelForWormsDetection=0
455 <br>LevelForSearchEnginesDetection=0
456 <br>LevelForFileTypesDetection=0
457 <br>ShowMenu=1
458 <br>ShowMonthStats=HB
459 <br>ShowDaysOfMonthStats=HB
460 <br>ShowDaysOfWeekStats=HB
461 <br>ShowHoursStats=HB
462 <br>ShowDomainsStats=0
463 rizwank 1.1 <br>ShowHostsStats=HBL
464 <br>ShowAuthenticatedUsers=0
465 <br>ShowRobotsStats=0
466 <br>ShowEMailSenders=HBML
467 <br>ShowEMailReceivers=HBML
468 <br>ShowSessionsStats=0
469 <br>ShowPagesStats=0
470 <br>ShowFileTypesStats=0
471 <br>ShowFileSizesStats=0
472 <br>ShowBrowsersStats=0
473 <br>ShowOSStats=0
474 <br>ShowOriginStats=0
475 <br>ShowKeyphrasesStats=0
476 <br>ShowKeywordsStats=0
477 <br>ShowMiscStats=0
478 <br>ShowHTTPErrorsStats=0
479 <br>ShowSMTPErrorsStats=1
480 <br>
481 </i>
482 </td></tr></table>
483 <br>
484 rizwank 1.1 Also don't forget that with Exchange, informations in a log analyses can't be exact.
485 Do not send any questions or requests for using AWStats with Exchange, this is not a problem in
486 AWStats and we have no time to support non opened products.<br>
487 If you want to have complete and accurate information with Exchange, forget using AWStats or use a
488 more serious mail serveur (Postfix, Sendmail, QMail...)<br>
489 <br>
490
491 <a name="MEDIASERVER"></a><br>
492 <b><u>FAQ-COM110 : SETUP FOR A MEDIA SERVER (REALMEDIA, WINDOWS MEDIA SERVER, DARWIN STREAMING SERVER)</u></b><br>
493 <font class=CProblem>PROBLEM:</font><br>
494 What do I have to do to use AWStats to analyze my Media Server log files.<br>
495 <font class=CSolution>SOLUTION:</font><br>
496 <br>
497 <u>For Realmedia</u><br>
498 <br>
499 Your log file will probably looks like this:<br>
500 <i>
501 216.125.146.50 - - [16/Sep/2002:14:57:21 -0500] "GET cme/rhythmcity/rcitycaddy.rm?cloakport=8080,554,7070 RTSP/1.0" 200 6672 [Win95_4.0_6.0.9.374_play32_NS80_en-US_586] [80d280e1-c9ae-11d6-fa53-d52aaed98681] [UNKNOWN] 281712 141 3 0 0 494<br>
502 </i>
503 <br>
504 Copy config awstats.model.conf file to "awstats.mediaserver.conf".
505 rizwank 1.1 Modify this new config file:
506 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
507 <i>
508 LogFile="/pathtomediaserverlog/mediaserverlog"
509 <br>LogType=S
510 <br>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %uabracket %other %other %other %other %other %other %other %other"
511 <br>LogSeparator="\s+"
512 <br>ShowMenu=1
513 <br>ShowMonthStats=UHB
514 <br>ShowDaysOfMonthStats=HB
515 <br>ShowDaysOfWeekStats=HB
516 <br>ShowHoursStats=HB
517 <br>ShowDomainsStats=HB
518 <br>ShowHostsStats=HBL
519 <br>ShowAuthenticatedUsers=0
520 <br>ShowRobotsStats=0
521 <br>ShowEMailSenders=0
522 <br>ShowEMailReceivers=0
523 <br>ShowSessionsStats=0
524 <br>ShowPagesStats=PB
525 <br>ShowFileTypesStats=HB
526 rizwank 1.1 <br>ShowFileSizesStats=0
527 <br>ShowBrowsersStats=1
528 <br>ShowOSStats=1
529 <br>ShowOriginStats=PH
530 <br>ShowKeyphrasesStats=0
531 <br>ShowKeywordsStats=0
532 <br>ShowMiscStats=0
533 <br>ShowHTTPErrorsStats=1
534 <br>ShowSMTPErrorsStats=0
535 </i>
536 </td></tr></table>
537 <br>
538 Now you can use AWStats as usual (run the update process and read statistics).<br>
539 <br>
540 <br>
541 <u>For Windows Media Server / Darwin Streaming Server</u><br>
542 <br>
543 1- If your Windows Media / Darwin streaming Server version allows it, setup your log format to write the following fields:
544 <i>
545 <br>c-ip
546 <br>date
547 rizwank 1.1 <br>time
548 <br>cs-uri-stem
549 <br>c-starttime
550 <br>x-duration
551 <br>c-rate
552 <br>c-status
553 <br>c-playerid
554 <br>c-playerversion
555 <br>c-playerlanguage
556 <br>cs(User-Agent)
557 <br>cs(Referer)
558 <br>c-hostexe
559 <br>c-hostexever
560 <br>c-os
561 <br>c-osversion
562 <br>c-cpu
563 <br>filelength
564 <br>filesize
565 <br>avgbandwidth
566 <br>protocol
567 <br>transport
568 rizwank 1.1 <br>audiocodec
569 <br>videocodec
570 <br>channelURL
571 <br>sc-bytes
572 </i>
573 <br>
574 To have the change effective, stop your server, remove old log files and restart the server.<br>
575 Listen to streaming files and check that your new log file looks like this:<br>
576 <i>
577 80.223.91.37 2002-10-08 14:18:58 mmst://mydomain.com/mystream 0 106 1 200 {F4A826EE-FA46-480F-A49B-76786320FC6B} 8.0.0.4477 fi-FI - - wmplayer.exe 8.0.0.4477 Windows_2000 5.1.0.2600 Pentium 0 0 20702 mms TCP Windows_Media_Audio_9 - - 277721
578 </i>
579 <br>
580 <br>
581 If your Windows Media/Darwin Streaming Server version does not allow to define your log format:<br>
582 Just follow instructions in step 2 directly but use the log format string found in first lines
583 of your log files (Just after the "<i>#Fields:</i>" string) as value for AWStats LogFormat
584 parameter. For example, you could have a LogFormat defined like this:<br>
585 <i>
586 LogFormat="c-ip date time c-dns cs-uri-stem c-starttime x-duration c-rate
587 c-status c-playerid c-playerversion c-playerlanguage cs(User-Agent)
588 cs(Referer) c-hostexe c-hostexever c-os c-osversion c-cpu filelength
589 rizwank 1.1 filesize avgbandwidth protocol transport audiocodec videocodec channelURL
590 sc-bytes c-bytes s-pkts-sent c-pkts-received c-pkts-lost-client
591 c-pkts-lost-net c-pkts-lost-cont-net c-resendreqs c-pkts-recovered-ECC
592 c-pkts-recovered-resent c-buffercount c-totalbuffertime c-quality s-ip s-dns
593 s-totalclients s-cpu-util"
594 </i>
595 <br>This means you don't use the AWStats tags but AWStats can often also understand all the IIS and/or
596 Windows Media Server tags.<br>
597
598 <br>
599 2- Then setup AWStats to analyze your Media Server log:<br>
600 Copy config awstats.model.conf file to "awstats.mediaserver.conf".<br>
601 Modify this new config file:
602 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
603 <i>
604 LogFile="/pathtomediaserver/mediaserverlog"
605 <br>LogType=S
606 <br>LogFormat="c-ip date time cs-uri-stem c-starttime x-duration c-rate c-status c-playerid c-playerversion c-playerlanguage cs(User-Agent) cs(Referer) c-hostexe c-hostexever c-os c-osversion c-cpu filelength filesize avgbandwidth protocol transport audiocodec videocodec channelURL sc-bytes"
607 <br>DecodeUA=1
608 <br>ShowMenu=1
609 <br>ShowMonthStats=UHB
610 rizwank 1.1 <br>ShowDaysOfMonthStats=HB
611 <br>ShowDaysOfWeekStats=HB
612 <br>ShowHoursStats=HB
613 <br>ShowDomainsStats=HB
614 <br>ShowHostsStats=HBL
615 <br>ShowAuthenticatedUsers=0
616 <br>ShowRobotsStats=0
617 <br>ShowEMailSenders=0
618 <br>ShowEMailReceivers=0
619 <br>ShowSessionsStats=0
620 <br>ShowPagesStats=PB
621 <br>ShowFileTypesStats=HB
622 <br>ShowFileSizesStats=0
623 <br>ShowBrowsersStats=1
624 <br>ShowOSStats=1
625 <br>ShowOriginStats=H
626 <br>ShowKeyphrasesStats=0
627 <br>ShowKeywordsStats=0
628 <br>ShowMiscStats=0
629 <br>ShowHTTPErrorsStats=1
630 <br>ShowSMTPErrorsStats=0
631 rizwank 1.1 </i>
632 </td></tr></table>
633 <br>
634 Now you can use AWStats as usual (run the update process and read statistics).<br>
635 <br>
636
637 <a name="ROTATE"></a><br>
638 <b><u>FAQ-COM120 : HOW TO ROTATE MY LOGS WITHOUT LOOSING DATA</u></b><br>
639 <font class=CProblem>PROBLEM:</font><br>
640 I want to archive/rotate my logs using my server system (for example logrotate) options or a third
641 software (rotatelog, cronolog) but I don't want to lose any visits information during the rotate
642 process.<br>
643 <font class=CSolution>SOLUTION:</font><br>
644 <li> If your config file is setup with a <a href="awstats_config.html#LogFile">LogFile</a> parameter
645 that point to your current running log file (required if you want to use the
646 <a href="awstats_config.html#AllowToUpdateStatsFromBrowser">AllowToUpdateStatsFromBrowser</a>
647 option to have "real-time" statistics), to avoid loosing too much records during the rotate
648 process, you must run the AWStats update JUST BEFORE the rotate process is done.<br>
649 The best way to do that on 'Linux like' OS is to use the linux built-in logrotate feature. You must
650 edit the logrotate config file used for your web server log file (usually stored in /etc/logrotate.d
651 directory) by adding the AWStats update process as a preprocessor command, like this example (bold
652 rizwank 1.1 lines are lines to add for having a prerotate process):<br><i>
653 /usr/local/apache/logs/*log<br>
654 {<br>
655 notifempty<br>
656 daily<br>
657 rotate 7<br>
658 compress<br>
659 <b>
660 sharedscripts<br>
661 prerotate<br>
662 /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=mydomainconfig<br>
663 endscript<br>
664 </b>
665 postrotate<br>
666 /usr/bin/killall -HUP httpd<br>
667 endscript<br>
668 }<br>
669 </i>
670 <br>
671 If using a such solution, this is sequential steps that happens:<br>
672 <table border=1 cellpadding=1 cellspacing=0 bgcolor=#F4F4F4 width="100%" class=CFAQ>
673 rizwank 1.1 <tr><td<b>Step</b></td><td><b>Description</b></td><td><b>Step name</b></td><td><b>Date/Time example</b></td></tr>
674 <tr valign=top><td>A</td><td>logrotate is started (by cron)</td><td>Start of logrotate</td><td>04:02:00</td></tr>
675 <tr valign=top><td>B</td><td><ul>awstats -update is launched by logrotate</td><td>Start of awstats</td><td>04:02:01</td><tr>
676 <tr valign=top><td>C</td><td><ul><ul>awstats start to read the log file <i>file.log</i></td><td> </td><td>04:02:02</td><tr>
677 <tr valign=top><td>D</td><td><ul>awstats has reached the end of log file so now it starts to save its database on disk.</td><td> </td><td>04:05:00</td><tr>
678 <tr valign=top><td>E</td><td><ul>awstats has finished to save its new database, so it stops</td><td>End of awstats</td><td>04:06:00</td><tr>
679 <tr valign=top><td>F</td><td>logrotate moves old log file <i>file.log</i> to a new name <i>file.log.sav</i>. Apache now logs in this file <i>file.log.sav</i> since log file handle has not been changed (only log file name has been renamed).</td><td>Log move</td><td>04:06:01</td><tr>
680 <tr valign=top><td>G</td><td>logrotate sends the -HUP or -USR1 signal to Apache.<br>With -HUP, Apache immediatly kills all its child process/thread, close log file <i>file.log.sav</i>, and reopen file <i>file.log</i>. So now, ALL hits are written to new file.<br>With -USR1, Apache only ask its child process/thread to stop only when HTTP request will be completely served. However it closes immediatly log file <i>file.log.sav</i>, and reopen file <i>file.log</i>. So only NEW hits are written to new log file. HTTP requests that are still running will write in old one.</td><td>Apache restart</td><td>04:06:02</td><tr>
681 <tr valign=top><td>H</td><td>logrotate starts compress the old log file <i>file.log.sav</i> into <i>file.log.gz</i></td><td>Start compress</td><td>04:06:03</td><tr>
682 <tr valign=top><td>I</td><td><ul>If some apache threads/processes are still running (because the kill sent was -USR1, so child process are waiting end of request before to stop), then those threads/processes are still writing to <i>file.log.sav</i>.<br>If kill -HUP was used, all process are already restarted so all writes in new <i>file.log</i>.</td><td> </td><td> </td><tr>
683 <tr valign=top><td>J</td><td>logrotate has finished to compress log file into <i>file.log.gz</i>. File <i>file.log.sav</i> is deleted.</td><td>End of compress<br>End of logrotate</td><td>04:07:03</td><tr>
684 <tr valign=top><td>K</td><td><ul>If signal was -USR1, some old childs can still run (when serving a very long request for example). So the log writing, still done in same file handle are going to a file that has been removed. So log writing are lost nowhere (this is only if -USR1 was used and if request was very long).</td><td> </td><td> </td><tr>
685 </table>
686 <br>
687 The advantage of this solution is that it is a very common way of working, used by a lot of
688 products, and easy to setup. You will notice that you can "lose" some hits:<br>
689 If you use the -HUP signal, you will only lose all hits that were written during D and E.
690 Note that you will also break all requests still running at G. In the example, it's a
691 1 minute lost (for small or medium web sites, it will be less than few seconds), so this
692 give you an error less than 0.07% (less for small web sites). This is not significant,
693 above all for a "statistics" progam.<br>
694 rizwank 1.1 If you use the -USR1 signal, you will not kill any request. But you will lose all hits that
695 were wrote during D and E (like with -HUP) but also all hits that are still running after H
696 (all very long request that requires several minutes to be served). If hit ends during I, it is
697 wrote in a log file already analyzed, if hit ends at K, it is wrote nowhere. In the example,
698 it's also a 0.07% error plus error for other not visible hits that were finished during I or K,
699 but number of such hits should be very low since only hits that started before G and not
700 finished after H are concerned. In most cases a hits needs only few milliseconds to be served
701 so lost hits could be ignored.<br>
702 <br>
703 Note also that if you have x logrotate config files, with each of them a postrotate with a
704 kill -HUP, you send a kill x times to your server process. So try to include several log files
705 in same logrotate config file. You can have several awstats update command in the same
706 prerotate section and you will send the -HUP only once, after all updates are finished.
707 However, doing this, you will have a lap time between D and F (were some hits are lost) that will
708 be higher.<br>
709 <br>
710
711 <li> Another common way of working is to choose to run the AWStats update process only once the log file has been
712 archived.<br>
713 This is required for example if you use the <a href="http://cronolog.org" target=awstatsbis>cronolog</a>
714 or rotatelog tools to rotate your log files. For example, Apache users can setup their Apache
715 rizwank 1.1 httpd config file to write log file through a pipe to cronolog or rotatelog using
716 Apache <i>CustomLog</i> directive:<br>
717 <i>CustomLog "|/usr/sbin/cronolog [cronolog_options] /var/logs/access.%Y%m%d.log" combined</i><br>
718 If you use a such feature, you can't trigger AWStats update process to be ran just BEFORE the
719 rotate is done, so you must run it AFTER the rotate process, so on the archived log file.<br>
720 To setup awstats to always point to last archive log file, you can use the 'tags' available for
721 <a href="awstats_config.html#LogFile">LogFile</a>.<br>
722 The problem with that is that your data are refreshed only after a rotate has done. However,
723 you will miss absolutely nothing (no hits) and your server processes are never killed.<br>
724 <br>
725 <li> So, if you really want to not lose absolutely no hit and want to have updates more
726 frequently than the rotate frequency, the best way is still an hybrid solution (i am not sure
727 that it worth the pain, and remember that statistics are only statistics):<br>
728 You run the awstats update process from you crontab frequently, every hour for example, and half and hour
729 before the rotate has done. See next FAQ to know how to setup a scheduled job.<br>
730 Then, once the rotate has been done (by the logrotate or by a piped cronolog log file), and
731 before the next scheduled awstats update process start,
732 you run another update process on the archived log file using the -logfile option to force
733 update on the archived log file and not the current log file defined in awstats
734 config file. This will allow you to update the half hour missing,
735 until the log rotate (AWStats will find the new lines). However don't forget that this
736 rizwank 1.1 particular update MUST be finished before the next croned update.<br>
737 <br>
738
739 <a name="CRONTAB"></a><br>
740 <b><u>FAQ-COM130 : HOW TO RUN AWSTATS UPDATE PROCESS FREQUENTLY</u></b><br>
741 <font class=CProblem>PROBLEM:</font><br>
742 AWStats must be ran frequently to update statistics. How can I do this ?<br>
743 <font class=CSolution>SOLUTION:</font><br>
744 A good way of working is to run the AWStats update process as a preprocessor of your log rotate
745 process. See previous FAQ (<a href="#ROTATE">FAQ-COM120</a>) for this.<br>
746 But you can also run AWStats update process regularly by a scheduler:<br>
747 <br>
748 <u>With Windows</u>, you can use the internal task scheduler.
749 The use of this tool is not an AWStats related problem, so please take a look at your Windows manual.
750 Warning, if you use <i>"awstats.pl -config=mysite -update"</i> in your scheduled task, you might
751 experience problem of failing task. Try this instead<br>
752 <i>"C:\WINNT\system32\CMD.EXE /C C:\[awstats_path]\awstats.pl -config=mysite -update"</i><br>
753 or<br>
754 <i>"C:\[perl_path]\perl.exe C:\[awstats_path]\awstats.pl -config=mysite -update"</i><br>
755 A lot of other scheduler (sharewares/freewares) are very good.<br>
756 <br>
757 rizwank 1.1 <u>With unix-like operating systems</u>, you can use the "<b>crontab</b>".<br>
758 This is examples of lines you can add in the cron file (see your unix reference manual for cron) :<br>
759 To run update every day at 03:50, use :<br>
760 <i>50 3 * * * /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=mysite -update >/dev/null</i><br>
761 To run update every hour, use :<br>
762 <i>0 * * * * /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=mysite -update >/dev/null</i><br>
763 <br>
764
765 <a name="EXCLUDEHOSTS"></a><br>
766 <b><u>FAQ-COM140 : HOW CAN I EXCLUDE MY IP ADDRESS (OR WHOLE SUBNET MASK) FROM STATS ?</u></b><br>
767 <font class=CProblem>PROBLEM:</font><br>
768 I don't want to see my own IP address in the stats or I want to exclude counting visits from a whole subnet.<br>
769 <font class=CSolution>SOLUTION:</font><br>
770 You must edit the config file to change the <a href="awstats_config.html#SkipHosts">SkipHosts</a> parameter.<br>
771 For example, to exclude:<br>
772 <li> your own IP address 123.123.123.123, use <a href="awstats_config.html#SkipHosts">SkipHosts</a>="123.123.123.123"<br>
773 <li> the whole subnet 123.123.123.xxx, use <a href="awstats_config.html#SkipHosts">SkipHosts</a>="REGEX[^123\.123\.123\.]"<br>
774 <li> all sub hosts xxx.myintranet.com, use <a href="awstats_config.html#SkipHosts">SkipHosts</a>="REGEX[\.myintranet\.com$]" (This one works only if DNS lookup is already done in your
775 log file).<br>
776 <br>
777
778 rizwank 1.1 <a name="EXTRA"></a><br>
779 <b><u>FAQ-COM145 : HOW TO USE THE EXTRA SECTIONS FEATURES ?</u></b><br>
780 <font class=CProblem>PROBLEM:</font><br>
781 I want to build personalized reports not provided in default AWStats reports. How can I setup
782 the Extra Sections parameters in my AWStats config file to do so ?<br>
783 <font class=CSolution>SOLUTION:</font><br>
784 Take a look at the <a href="awstats_extra.html">Using AWStats Extra Sections features</a><br>
785 <br>
786
787 <a name="BENCHMARK"></a><br>
788 <b><u>FAQ-COM150 : BENCHMARK / FREQUENCY TO LAUNCH AWSTATS TO UPDATE STATISTICS</u></b><br>
789 <font class=CProblem>PROBLEM:</font><br>
790 What is AWStats speed ?<br>
791 What is the frequency to launch AWStats process to update my statistics ?<br>
792 <font class=CSolution>SOLUTION:</font><br>
793 All benchmarks information and advice on frequency for update process are related into
794 the <a href="awstats_benchmark.html">Benchmark page</a>.<br>
795 <br>
796
797 <a name="DNS"></a><br>
798 <b><u>FAQ-COM200 : HOW REVERSE DNS LOOKUP WORKS, UNRESOLVED IP ADDRESSES</u></b><br>
799 rizwank 1.1 <font class=CProblem>PROBLEM:</font><br>
800 The reported page AWStats shows me has no hostnames, only IP addresses, countries reported are all "unknown".<br>
801 <font class=CSolution>SOLUTION:</font><br>
802 When AWStats find an IP address in your log file, it tries a reverse DNS lookup to find
803 the hostname and domain if the <a href="awstats_tools.html#DNSLookup">DNSLookup</a> parameter, in your AWStats config file, is <a href="awstats_config.html#DNSLookup">DNSLookup</a>=1
804 (Default value). So, first, check if you have the good value. The <a href="awstats_config.html#DNSLookup">DNSLookup</a>=0
805 must be used only if your log file contains already resolved IP address. For example,
806 when you set up Apache with the <i>HostNameLookups=on</i> directive. When you
807 ask your web server to make itself the reverse DNS lookup to log hostname instead
808 of IP address, you will still find some IP addresses in your log file because
809 the reverse DNS lookup is not always possible. But if your web server fails in
810 it, AWStats will also fails (All reverse DNS lookups use the same system API).
811 So to avoid AWStats to make an already done lookup (with success or not), you
812 can set <a href="awstats_config.html#DNSLookup">DNSLookup</a>=0 in AWStats config file.
813 If you prefer, you can make the reverse DNS lookup on a log file before running
814 your log analyzer (If you only need to convert a logfile with IP Addresses into a
815 logfile with resolved hostnames). You can use for this <a href="awstats_tools.html#logresolvemerge">logresolvemerge</a> tool
816 provided with AWStats distribution (This tools is an improved version of <i>logresolve</i> provided with Apache).<br>
817 <br>
818
819 <a name="DIFFERENT_RESULTS"></a><br>
820 rizwank 1.1 <b><u>FAQ-COM250 : DIFFERENT RESULTS THAN OTHER ANALYZER</u></b><br>
821 <font class=CProblem>PROBLEM:</font><br>
822 I also use Webalizer, Analog (or another log analyzer) and it doesn't report the same results than AWStats. Why ?<br>
823 <font class=CSolution>SOLUTION:</font><br>
824 If you compare AWStats results with an other log file analyzer, you will found some differences,
825 sometimes very important. In fact, all analyzer (even AWStats) make "over reporting" because of the
826 problem of proxy-servers and robots. However AWStats is one of the most accurate and its "over reporting"
827 is very low where all other analyzers, even the most famous, have a VERY HIGH error rate (10% to 200% more than reality !).<br>
828 <b>This is the most important reasons why you can find important differences:</b><br>
829 <li> Some dynamic pages generated by CGI programs are not counted by some analyzer (ie Webalizer) like
830 a "Page" (but only like a "Hit") if CGI prog does not end with a defined extension (.cgi, ...), so they are not included
831 correctly in their statistics. AWStats use on oposite policy, assuming a file is a page except if
832 type is in a list (See <a href="awstats_config.html#NotPageList">NotPageList</a> parameter). Error rate
833 with a such policy is lower.<br>
834 <li> AWStats is able to detect robots visits. Most analyzers think robots visits are human visitors.
835 This error make them to report more visits and visitors than reality.
836 When AWStats reports a "1 visitor", it means "1 human visitor" (even if it's not posible to detect
837 all robots, most of them are detected). "Robots visitors" are reported separately in the "Robots/Spiders visitors" chart.<br>
838 <li> Some log analyzers use the "Hits" to count visitors. This is a very bad way of working :
839 Some visitors use a lot of proxy servers to surf (ie: AOL users), this means it's possible that several
840 hosts (with several IP addresses) are used to reach your site for only one visitor (ie: one proxy server download
841 rizwank 1.1 the page and 2 other servers download all images). Because of this, if stats of unique visitors are made on "hits",
842 3 users are reported but it's wrong. So AWStats considers only HTML "Pages" to count unique visitors.
843 This decrease the error, not completely, because it's always possible that a proxy server download one HTML frame and
844 another one download another frame, but this make the over-reporting of unique visitors less important.<br>
845 <li> Another important reason to have difference is that an error log files is not always completely sorted
846 but only "nearly" sorted because of cache and writing log engines used by server. Nearly all log
847 analyzers (commercial and not) assumes that log file is "exactly" sorted by hit date to calculate
848 visits, entry and exit pages. But there is nothing that guaranties this and some log files
849 are only "nearly" sorted, above all log files on highly loaded servers.
850 AWStats has an advanced parsing algorithm that is able to count
851 correctly visits, entry and exit pages even if log file is only "nearly" sorted.<br>
852 <li> Then, there is internal bugs in log analyzers that make reports wrong.
853 For example, a lot of users have reported that Webalizer "doubles" the number of visits or visitors
854 in some circumstances.<br>
855 <b>There is also other reasons, however those points explains only small differences:</b><br>
856 <li> To differenciate new visits of a same visitor, log analyers uses a visit time-out. If value differs,
857 then results differ (on visit count and entry and exit pages).
858 A such time-out is a fixed value (For example 60 minutes) meaning if a visitor make a hit
859 59 minutes after downloading the previous page, it's the same visits, if he make it 61 minutes after, it's a new visit.
860 Of course, there is no realy difference between 59 and 61, but couting visits without
861 time-out is not possible. And because the most important is to have a time-out (and not
862 rizwank 1.1 really it's value), AWStats time-out is not an "exact" value but is "around" 60 minutes.
863 This allows AWStats to have better speed processing time, so you also might experience
864 little differences, in visit count, between AWStats and another log analyzer even if
865 their time-out are both defined to same value (because AWStats time-out is not exactly
866 but nearly value defined).<br>
867 <li> There is also differences in log analyzers databases and algorithms that make details of results less or more accurate:<br>
868 AWStats has a larger browsers, os', search engines and robots database, so reports concerning this are more accurate.<br>
869 AWStats has url syntax rules to find keywords or keyphrases used to find your site, but AWStats has also
870 an algorithm to detect keywords of unknown search engines with unknown url syntax rule.<br>
871 AWStats does not count twice (by default) redirects made by rewrite rules that makes two hits into
872 log files but that are only one page "viewed".<br>
873 Etc...<br>
874 <br>
875 If you want to check how serious is your log analyzer, try to parse the following log file.
876 It's a very common log file but results will show you how bad are most log analyzers (above
877 all commercial products):<br>
878 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
879 <i><font style="font: 8px verdana,arial,helvetica">
880 <br># This is a sample of log file that contains a lot of various data we can find
881 <br># in a log file. Great sample to test reliability and accuracy of any log
882 <br># analyzer.
883 rizwank 1.1 <br># ----------------------------------------------------------------------------
884 <br># This sample log file contains 10 differents IPs that are :
885 <br># 6 human visits, done by 5 different true visitors
886 <br># 1 proxy visit, done by one of the 5 true visitors
887 <br># 1 try of a 6th human visit failed because of wrong url (He saw nothing)
888 <br># 1 bot visit to index your site
889 <br># 1 worm attack
890 <br># 1 add to favourites (two hits but first is non root hit with error meaning it's same "add")
891 <br># ----------------------------------------------------------------------------
892 <br># 80.8.55.1 2 visits (start at 00:00:00 and at 12:00:00 with both entry page on /)
893 <br># 80.8.55.2 this is not a visit, only an image included into a page of an other site
894 <br># 80.8.55.3 1 visit (and add home page to favourites)
895 <br># 80.8.55.4 same visitor than 80.8.55.3 using aol proxy
896 <br># 80.8.55.5 not a visit (but a bot indexing)
897 <br># 80.8.55.6 1 visit (authenticated visitor)
898 <br># 80.8.55.7 1 visit (authenticated visitor with space in name)
899 <br># 80.8.55.8 not a visit (try but failed twice with 404 and 405 error)
900 <br># 80.8.55.9 not a visit (but a worm attack)
901 <br># 80.8.55.10 1 visit that come from a web page that is not a search engine
902 <br>
903 <br>80.8.55.1 - - [01/Jan/2001:00:00:10 +0100] "GET /page1.html HTTP/1.0" 200 7009 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
904 rizwank 1.1 <br>80.8.55.1 - - [01/Jan/2001:00:00:00 +0100] "GET / HTTP/1.0" 200 7009 "http://www.sitereferer/cgi-bin/search.pl?q=a" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
905 <br>80.8.55.1 - - [01/Jan/2001:00:00:20 +0100] "GET /page2.cgi HTTP/1.0" 200 7009 "http://localhost/page1.html" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
906 <br>80.8.55.1 - - [01/Jan/2001:00:00:25 +0100] "GET /page3 HTTP/1.0" 200 7009 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
907 <br>80.8.55.1 - - [01/Jan/2001:00:00:30 +0100] "GET /image.gif HTTP/1.0" 200 7009 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
908 <br>80.8.55.1 - - [01/Jan/2001:00:00:35 +0100] "GET /image2.png HTTP/1.0" 200 7009 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
909 <br>80.8.55.1 - - [01/Jan/2001:00:00:40 +0100] "GET /dir/favicon.ico HTTP/1.0" 404 299 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
910 <br>80.8.55.1 - - [01/Jan/2001:00:00:40 +0100] "GET /favicon.ico HTTP/1.0" 200 299 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
911 <br>80.8.55.1 - - [01/Jan/2001:12:00:00 +0100] "GET / HTTP/1.0" 200 7009 "http://WWW.SiteRefereR:80/cgi-bin/azerty.pl?q=a" "Mozilla/4.7 [fr] (Win95; I)"
912 <br>80.8.55.1 - - [01/Jan/2001:12:00:10 +0100] "GET /page1.html HTTP/1.0" 200 7009 "-" "Mozilla/4.7 [fr] (Win95; I)"
913 <br>80.8.55.1 - - [01/Jan/2001:12:00:20 +0100] "GET /page2.cgi HTTP/1.0" 200 7009 "-" "Mozilla/4.7 [fr] (Win95; I)"
914 <br>80.8.55.1 - - [01/Jan/2001:12:00:25 +0100] "GET /page3 HTTP/1.0" 200 7009 "-" "Mozilla/4.7 [fr] (Win95; I)"
915 <br>80.8.55.1 - - [01/Jan/2001:12:00:30 +0100] "GET /image.gif HTTP/1.0" 200 7009 "-" "Mozilla/4.7 [fr] (Win95; I)"
916 <br>80.8.55.1 - - [01/Jan/2001:12:00:35 +0100] "GET /image2.png HTTP/1.0" 200 7009 "-" "Mozilla/4.7 [fr] (Win95; I)"
917 <br>80.8.55.1 - - [01/Jan/2001:12:00:40 +0100] "GET /js/awstats_misc_tracker.js HTTP/1.1" 200 4998 "-" "Mozilla/4.7 [fr] (Win95; I)"
918 <br>80.8.55.1 - - [01/Jan/2001:12:00:45 +0100] "GET /js/awstats_misc_tracker.js?SCREEN=1024x768&CDI=32&JAVA=true&UC=UserCode1056710428488r6832&SC=SessionCode1056710428488r6832&SHK=N&FLA=Y&RP=N&MOV=N&WMA=Y&PDF=Y HTTP/1.1" 200 4998 "-" "Mozilla/4.7 [fr] (Win95; I)"
919 <br>
920 <br>80.8.55.2 - - [01/Jan/2001:12:01:00 +0100] "GET /hitfromothersitetoimage.gif HTTP/1.0" 200 7009 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3) Gecko/20030312"
921 <br>
922 <br>80.8.55.3 - - [01/Jan/2001:12:01:10 +0100] "GET / HTTP/1.0" 200 7009 "http://www.sitereferer:81/cgi-bin/azerty.pl" "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5a) Gecko/20030728 Mozilla Firebird/0.6.1"
923 <br>80.8.55.3 - - [01/Jan/2001:12:01:15 +0100] "GET /page1.html HTTP/1.0" 200 7009 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5a) Gecko/20030728 Mozilla Firebird/0.6.1"
924 <br>80.8.55.3 - - [01/Jan/2001:12:01:20 +0100] "GET /page2.cgi?x=a&family=a&y=b&familx=x HTTP/1.0" 200 7009 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5a) Gecko/20030728 Mozilla Firebird/0.6.1"
925 rizwank 1.1 <br>80.8.55.3 - - [01/Jan/2001:12:01:25 +0100] "GET /page3 HTTP/1.0" 200 7009 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5a) Gecko/20030728 Mozilla Firebird/0.6.1"
926 <br>80.8.55.3 - - [01/Jan/2001:12:01:30 +0100] "GET /image.gif HTTP/1.0" 200 7009 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5a) Gecko/20030728 Mozilla Firebird/0.6.1"
927 <br>80.8.55.3 - - [01/Jan/2001:12:01:35 +0100] "GET /image2.png HTTP/1.0" 200 7009 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5a) Gecko/20030728 Mozilla Firebird/0.6.1"
928 <br>
929 <br>80.8.55.4 - - [01/Jan/2001:12:01:45 +0100] "GET /samevisitorthan80.8.55.3usingaolproxy.gif HTTP/1.0" 200 7009 "-" "Mozilla/3.0 (Windows 98; U) Opera 6.03"
930 <br>
931 <br>80.8.55.5 - - [01/Jan/2001:12:02:00 +0200] "GET /robots.txt HTTP/1.0" 200 299 "-" "GoogleBot"
932 <br>80.8.55.5 - - [01/Jan/2001:12:02:05 +0200] "GET / HTTP/1.0" 200 7009 "-" "GoogleBot"
933 <br>
934 <br>80.8.55.6 - john [01/Jan/2001:13:00:00 +0100] "GET /cgi-bin/order.cgi?x=a&family=a&productId=998&titi=i&y=b&y=b HTTP/1.0" 200 7009 "http://www.google.com/search?sourceid=navclient&ie=utf-8&oe=utf-8&q=ma%C3%AEtre" "SAGEM-myX-5m/1.0_UP.Browser/6.1.0.6.1.103_(GUI)_MMP/1.0_(Google_WAP_Proxy/1.0)"
935 <br>80.8.55.6 - john [01/Jan/2001:13:00:00 +0100] "GET /cgi-bin/order.cgi?x=a&family=a&productId=998&titi=i&y=b&y=b HTTP/1.0" 200 7009 "http://www.google.com/search?sourceid=navclient&ie=utf-8&oe=utf-8&q=élève" "SAGEM-myX-5m/1.0_UP.Browser/6.1.0.6.1.103_(GUI)_MMP/1.0_(Google_WAP_Proxy/1.0)"
936 <br>
937 <br>80.8.55.7 - John Begood [01/Jan/2001:13:01:00 +0100] "GET /cgi-bin/order.cgi;family=f&type=t&productId=999&titi=i#BIS HTTP/1.0" 200 7009 "-" "Mozilla/3.01 (compatible;)"
938 <br>
939 <br>80.8.55.8 - - [01/Jan/2001:14:01:20 +0100] "GET /404notfoundpage.html?paramnotpagefound=valparamnotpagefound HTTP/1.0" 404 0 "http://refererto404nofoundpage/pageswithbadlink.html?paramrefnotpagefound=valparamrefnotpagefound" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
940 <br>80.8.55.8 - - [01/Jan/2001:14:01:20 +0100] "GET /405error.html HTTP/1.0" 405 0 "http://refererto405error/pagesfrom405.html" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
941 <br>
942 <br>80.8.55.9 - - [01/Jan/2001:15:00:00 +0200] "GET /default.ida?XXXXXXXXXXXXXXXXXX%u6858%ucbd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a HTTP/1.0" 404 299 "-" "-"
943 <br>
944 <br>80.8.55.10 - - [01/Jan/2001:16:00:00 -0300] "GET / HTTP/1.1" 200 70476 "http://us.f109.mail.yahoo.com/ym/ShowLetter?box=Inbox&MoreYahooParams..." "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; Hotbar 4.2.8.0)"
945 <br>80.8.55.10 - - [01/Jan/2001:17:00:00 -0300] "GET /page1.html HTTP/1.1" 200 70476 "http://www.freeweb.hu/icecat/filmek/film04.html" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; Hotbar 4.2.8.0)"
946 rizwank 1.1 <br>80.8.55.10 - - [01/Jan/2001:18:00:00 -0300] "GET /cgi-bin/awredir.pl?url=http://xxx.com/aa.html HTTP/1.1" 302 70476 "http://us.f109.mail.yahoo.com/ym/ShowLetter?box=Inbox&MoreYahooParams..." "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; Hotbar 4.2.8.0)"
947 </i>
948 </td></tr></table><br>
949 <u>This is what you should find:</u><br>
950 6 true human visits<br>
951 5 different true visitors<br>
952 1 bot visit<br>
953 1 worm attack<br>
954 The entry pages for true visits should be "/" (even for 80.8.55.1) or "/cgi-bin/order.cgi" but nothing else.<br>
955 Note: I did not find any commercial log analyzer that can deal such a common log file correctly, so if you find, let me know !<br>
956 <br>
957
958 <a name="DIFFERENCE_HOURS"></a><br>
959 <b><u>FAQ-COM300 : DIFFERENCE BETWEEN LOCAL HOURS AND AWSTATS REPORTED HOURS</u></b><br>
960 <font class=CProblem>PROBLEM:</font><br>
961 I use IIS and there's a difference between local hour and AWStats reported hour. For example I made a hit on a page
962 at 4:00 and AWStats report I hit it at 2:00.<br>
963 <font class=CSolution>SOLUTION:</font><br>
964 This is not a problem of time in your local client host.
965 AWStats use only time reported in logs by your server
966 and all time are related to server hour. The problem is that IIS in some foreign
967 rizwank 1.1 versions puts GMT time in its log file (and not local time). So, you have also
968 GMT time in your statistics.<br>
969 You can wait that Microsoft change this in next IIS versions.
970 However, Microsoft sheet Q271196 "IIS Log File Entries Have the Incorrect Date and Time Stamp" says: <br>
971 <i>The selected log file format is the W3C Extended Log File
972 Format. The extended log file format is defined in the W3C
973 Working Draft WD-logfile-960323 specification by Phillip
974 M. Hallam-Baker and Brian Behlendorf. This document defines
975 the Date and Time files to always be in GMT. This behavior
976 is by design.</i><br>
977 So this means this way of working might never be changed, so another chance is to use the AWStats plugin 'timezone'.
978 Warning, this plugin need the perl module Time::Local and it reduces seriously AWStats speed.<br>
979 To enable the plugin, uncomment the following line in your config file.<br>
980 <i>LoadPlugin="timezone TZ"</i><br>
981 where TZ is value of your signed timezone (+2 for Paris, -8 for ...)<br>
982 <br>
983
984 <a name="OLDLOG"></a><br>
985 <b><u>FAQ-COM350 : HOW CAN I PROCESS AN OLD LOG FILE ?</u></b><br>
986 <font class=CProblem>PROBLEM:</font><br>
987 I want to process an old log file to include its data in my AWStats reports.<br>
988 rizwank 1.1 <font class=CSolution>SOLUTION:</font><br>
989 You must change your <a href="awstats_config.html#LogFile">LogFile</a> parameter to point to the old log file and run
990 the update (or use the -LogFile option on command line to overwrite <a href="awstats_config.html#LogFile">LogFile</a> parameter).
991 The update process can only accept files in chronological order for a particular month, so if you
992 have already processed a recent file and forgot to run update
993 on a log file that contains older data, you must before reset all
994 your statistics (see <a href="#FAQ-COM500">FAQ-COM500</a>) and restart all the update processes
995 for all past log files and in chronological order.<br>
996 However, there is a "tip" that allow you to rebuild only the month were you missed data:<br>
997 Imagine we are on 5th of July 2003, all your statistics are up to date except for the 10th of
998 April 2003 (you forgot to run the update process for this day, so there is no visit for this
999 day). You can :<br>
1000 - Reset the statistics for April only (this means remove the file <i>awstats042003.[config.]txt</i>
1001 as explained in <a href="#FAQ-COM500">FAQ-COM500</a>),<br>
1002 - Move the statistics history files for month after April (file <i>awstats052003.[config.]txt</i>,
1003 <i>awstats062003.[config.]txt</i>,...) into a temp directory (so that it is no more in DirData
1004 directory as if they were deleted).<br>
1005 - Run update process on all log files for April (in chronological order). AWStats does not
1006 complain about "too old record" because there is no history files in DirData directory
1007 that contains compiled data more recent than records into log you process.<br>
1008 - Moved back the month history files you saved into your DirData directory.<br>
1009 rizwank 1.1 Your statistics are up to date and the missing days are no more missing.<br>
1010 <br>
1011
1012 <a name="MULTILOG"></a><br>
1013 <b><u>FAQ-COM360 : HOW CAN I PROCESS SEVERAL LOG FILES IN ONE RUN ?</u></b><br>
1014 <font class=CProblem>PROBLEM:</font><br>
1015 How can I update my statistics for several log file, in one run ?<br>
1016 <font class=CSolution>SOLUTION:</font><br>
1017 A solution should be to setup your config file with something like:<br>
1018 <i>LogFile=mylog*.log</i><br>
1019 However, with such a syntax, AWStats can't know in wich order processing log files (wich log file is the first, next or last). So
1020 to work like this you must use the following syntax:<br>
1021 <i>LogFile="/pathto/logresolvemerge.pl mylog*.log |"</i><br>
1022 <a href="awstats_tools.html#logresolvemerge">Logresolvemerge</a> is a tool provided with
1023 AWStats (in tools directory) that merge several log files on the fly sending line by line always the older record
1024 from a list of several log files. Using such a tool as a pipe source for AWStats <a href="awstats_config.html#LogFile">LogFile</a>
1025 parameter is a very good solution because, it allows you to merge log files whatever their size
1026 with no memory use, no hard disk use (no temporary files built), it is fast, it prevents
1027 you from a bad order if your log files are not correctly ordered, etc...<br>
1028 This tool can also be used to process log files from load balanced systems (see <a href="awstats_faq.html#LOADLOG">FAQ-COM400</a>)<br>
1029 <br>
1030 rizwank 1.1
1031 <a name="LOADLOG"></a><br>
1032 <b><u>FAQ-COM400 : HOW CAN I UPDATE MY STATISTICS WHEN I USE A LOAD BALANCING SYSTEM THAT SPLITS MY LOGS ?</u></b><br>
1033 <font class=CProblem>PROBLEM:</font><br>
1034 How can I update my statistics when i use a load balancing system that split my logs ?<br>
1035 <font class=CSolution>SOLUTION:</font><br>
1036 First solution is to merge all split log files resulted from all your load balanced servers into one. For this, you can use
1037 the <a href="awstats_tools.html#logresolvemerge">logresolvemerge</a> tool provided with AWStats :<br>
1038 <i>logresolvemerge.pl file1.log file2.log ... filen.log > newfiletoprocess.log</i><br>
1039 And setup the <a href="awstats_config.html#LogFile">LogFile</a> parameter in your config file to process the <i>newfiletoprocess.log</i> file or use
1040 the <i>-LogFile</i> command line option to overwrite <a href="awstats_config.html#LogFile">LogFile</a> value.<br>
1041 As an other solution, if you miss disk space, or to save time, you can ask <a href="awstats_tools.html#logresolvemerge">logresolvemerge</a>
1042 to merge log files on the fly during the AWStats update process. For this, you can use the following syntax in your AWStats config file:<br>
1043 <i>LogFile="/pathto/logresolvemerge.pl file*.log |"</i><br>
1044 See also <a href="awstats_faq.html#MULTILOG">FAQ-COM360</a> for explanation on logresolvemerge use.<br>
1045 <br>
1046
1047 <a name="RESET"></a><br>
1048 <b><u>FAQ-COM500 : HOW CAN I RESET ALL MY STATISTICS ?</u></b><br>
1049 <font class=CProblem>PROBLEM:</font><br>
1050 I want to reset all my statistics to restart the update process from the beginning.<br>
1051 rizwank 1.1 <font class=CSolution>SOLUTION:</font><br>
1052 All analyzed data are stored by AWStats in history files called <i>awstatsMMYYYY.[config.]txt</i> (one file each month).
1053 You will find those files in directory defined by <a href="awstats_config.html#DirData">DirData</a> parameter (same directory than awstats.pl by default).<br>
1054 To reset all your statistics, just delete all files <i>awstatsMMYYYY.txt</i><br>
1055 To reset all your statistics built for a particular config file, just delete all files <i>awstatsMMYYYY.myconfig.txt</i><br>
1056 Warning, if you delete those data files, you won't be able to recover your stats back, unless you
1057 kept old log files somewhere. You will have to process all past log files (in chronological order)
1058 to get your statistics back.<br>
1059 <br>
1060
1061 <a name="DAILY"></a><br>
1062 <b><u>FAQ-COM600 : HOW CAN I COMPILE AND BUILD STATISTICS ON A DAILY BASIS ONLY ?</u></b><br>
1063 <font class=CProblem>PROBLEM:</font><br>
1064 How can I compile and build statistics on a daily basis. I mean i want to have a full report
1065 with all charts with data for a particular day only and want one report for each day of month.<br>
1066 <font class=CSolution>SOLUTION:</font><br>
1067 This is an non documented and not supported trick, as this is not the standard way of working:<br>
1068 First, run the update process at midnight (or on a log file that was rotated at midnight so that
1069 it contains only data for this particular day (you can choose another hour in night if you want
1070 to have days that "start" at an different hour).<br>
1071 Once the update process has been ran, MOVE (and not copy) the history file built by AWStats. For
1072 rizwank 1.1 example on Unix like systems:<br>
1073 <i>mv mydirdata/awstatsMMYYYY.mydomain.txt mydirdate/awstatsDDMMYYYY.mydomain.txt</i><br>
1074 Note that the name has been changed by adding the day. Repeat this each day after the update process.<br>
1075 With this you will have one history file for each day. You can then see full stats for
1076 a particular day by adding the non documented parameter -day=DD on command line (with others
1077 like -month=MM and -year=YYYY). If ran from a browser you can also add &day=DD on URL.<br>
1078 However, if you have full day by day statistics, you don't have anymore statistics for full month,
1079 except if you create a second config file that whose history files would not be moved.<br>
1080 <br>
1081
1082 <a name="EDITHISTORY"></a><br>
1083 <b><u>FAQ-COM700 : CAN I SAFELY REMOVE A LINE IN HISTORY FILES (awstatsMMYYYY*.txt) ?</u></b><br>
1084 <font class=CProblem>PROBLEM:</font><br>
1085 After processing a log file I want to change my statistics
1086 without running AWStats update process but changing directly data in AWStats historical database files.<br>
1087 <font class=CSolution>SOLUTION:</font><br>
1088 If you remove a lines starting with "BEGIN_" or "END_", AWStats will find your file "corrupted" so you must not change those
1089 two kinds of lines.<br>
1090 You can change, add or remove any line that is in any sections but if you do this, you must
1091 also update the MAP section (lines between BEGIN_MAP and END_MAP) because this section contains the offset in
1092 file of each other sections for direct I/O access.
1093 rizwank 1.1 If history file is the last one, you can easily do that by removing completely the MAP section and run an update process.
1094 Like that AWStats will rewrite the history file and the MAP section will be rewritten (MAP section is not read by
1095 update process, only written). You do this at your own risk. The main risk is that some charts will report wrong values
1096 or be unavailable.<br>
1097 <br>
1098
1099
1100 <hr><br>
1101
1102
1103 <a name="MISSINGDOLLAR"></a><br>
1104 <b><u>FAQ-SET050 : ERROR "MISSING $ ON LOOP VARIABLE ..."</u></b><br>
1105 <font class=CProblem>PROBLEM:</font><br>
1106 When I run awstats.pl from command line, I get:<br>
1107 <i>"Missing $ on loop variable at awstats.pl line xxx"</i><br>
1108 <font class=CSolution>SOLUTION:</font><br>
1109 Problem is in your Perl interpreter. Try to install or reinstall a more recent/stable Perl interpreter.<br>
1110 You can get new Perl version at <a href="http://www.activestate.com/ActivePerl/">ActivePerl</a> (<font color=#221188>Win32</font>)
1111 or <a href="http://www.perl.com/pub/language/info/software.html">Perl.com</a> (<font color=#221188>Unix/Linux/Other</font>).<br>
1112 <br>
1113
1114 rizwank 1.1 <a name="CGISOURCE"></a><br>
1115 <b><u>FAQ-SET100 : I SEE PERL SCRIPT'S SOURCE INSTEAD OF ITS EXECUTION</u></b><br>
1116 <font class=CProblem>PROBLEM:</font><br>
1117 When I try to execute the Perl script through the web server,
1118 I see the Perl script's source instead of the HTML result page of its execution !<br>
1119 <font class=CSolution>SOLUTION:</font><br>
1120 This is not a problem of AWStats but a problem in your web server setup.
1121 awstats.pl file must be in a directory defined in your web server to be a "cgi" directory,
1122 this means, a directory configured in your web server to contain "executable" files and
1123 not to documents files.
1124 You have to read your web server manual to know how to setup a directory to be an
1125 "executable cgi" directory (With IIS, you have some checkbox to check in directory
1126 properties, with Apache you have to use the "ExecCGI" option in the directory "Directive").<br>
1127 <br>
1128
1129 <a name="SPAWNERROR"></a><a name="INTERNAL"></a><br>
1130 <b><u>FAQ-SET150 : INTERNAL ERROR 500 IN MY BROWSER</u></b><br>
1131 <b><u>FAQ-SET200 : ERROR "... COULDN'T CREATE/SPAWN CHILD PROCESS..."</u></b><br>
1132 <font class=CProblem>PROBLEM:</font><br>
1133 AWStats seems to run fine at the command prompt but when ran as a CGI from a browser, I get an <i>"Internal Error 500"</i>.<br>
1134 I also also might have the following message in my Apache error log file (or in browser with Apache 2.0+):<br>
1135 rizwank 1.1 <i>...couldn't create/spawn child process: c:/mywebroot/cgi-bin/awstats.pl</i><br>
1136 <font class=CSolution>SOLUTION:</font><br>
1137 First, try to run awstats.pl from command line to see if file is correct. If you get some
1138 syntax errors and use a Unix like OS, check if your file is a Unix like text file (This
1139 means each line end with a LF char and not a CR+LF char).<br>
1140 If awstats.pl file runs correctly from command line, this is probably because your
1141 web server is not able to known how to run perl scripts. This problem can occur with Apache web
1142 servers with no internal Perl interpreter (mod_perl not active). To solve this, you must
1143 tell Apache where is your external Perl interpreter.<br>
1144 <u>For this, you have 2 solutions:</u><br>
1145 1) Add the following directive in your Apache <b>httpd.conf</b> config (or remove the # to uncomment it if line is already available)<br>
1146 <b><i>ScriptInterpreterSource registry</i></b><br>
1147 Then restart Apache. This will tell Apache to look into the registry to find the program associated to .pl extension.<br>
1148 2) Other solution (not necessary if first solution works): Change the first line of awstats.pl file with the full path of your Perl interpreter.<br>
1149 Example with Windows OS and ActivePerl Perl interpreter (installed in C:\Program Files\ActiveState\ActivePerl),
1150 you must change the first line of awstats.pl file with:<br>
1151 <i>#!c:/program files/activestate/activeperl/bin/perl</i><br>
1152 <br>
1153
1154 <a name="CRASH"></a><a name="CRASH"></a><br>
1155 <b><u>FAQ-SET220 : CRASH WHILE RUNNING AWSTATS.PL OR PAGE CONTENT ONLY PARTIALY LOADED ON WINDOWS XP</u></b><br>
1156 rizwank 1.1 <font class=CProblem>PROBLEM:</font><br>
1157 Sometimes my browser (Most often IE6) crash while running awstats.pl with some AWStats configuration.
1158 With some other versions or browsers, page content is partialy loaded.<br>
1159 <font class=CSolution>SOLUTION:</font><br>
1160 Problem was with WinXP and WinXPpro as documented at MS site Q317949;<br>
1161 "Socket Sharing Creates Data Loss When Listen and Accept Occur on Different Processes"<br>
1162 Result was that MSIE would crash or display nothing. Netscape and Opera handled the socket better but displayed the pages partially.<br>
1163 The effect of the bug was more prononced as the page contents increased (above 30k).<br>
1164 <a href="http://support.microsoft.com/default.aspx?scid=kb;EN-US;q317949">http://support.microsoft.com/default.aspx?scid=kb;EN-US;q317949</a><br>
1165 And also at Apache.org<br>
1166 <a href="http://www.apache.org/dist/httpd/binaries/win32/">http://www.apache.org/dist/httpd/binaries/win32/</a><br>
1167 MS produced a Hotfix which is now included in SP1.<br>
1168 But the best solution is to use a better web browser. Take a look at <a href="http://www.mozilla.org/products/firefox/">Firefox</a>,
1169 one of the best and most popular web browser.<br>
1170 <br>
1171
1172 <a name="PERSONALIZEDLOG"></a><br>
1173 <b><u>FAQ-SET250 : LOG FORMAT SETUP OR ERRORS</u></b><br>
1174 <font class=CProblem>PROBLEM:</font><br>
1175 Which value do I have to put in the LogFormat parameter to make AWStats working with my log
1176 file format ?<br>
1177 rizwank 1.1 <font class=CSolution>SOLUTION:</font><br>
1178 The AWStats config file give you all possible values for LogFormat parameter.
1179 To help you, this is some common cases of log file format, and
1180 the corresponding value for LogFormat you must use in your AWStats config file:<br>
1181
1182 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
1183 <u>If your log records are EXACTLY like this (<b>NCSA combined/XLF/ELF</b> log format):</u><br>
1184 <i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 1234 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"</i><br>
1185 You must use : <i>LogFormat=1</i><br>
1186 This is same than: <i>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"</i><br>
1187 </td></tr></table>
1188
1189 <u>If your log records are EXACTLY like this (<b>NCSA combined with several virtualhostname</b> sharing same log file).</u><br>
1190 <i>virtualserver1 62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 1234 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"</i><br>
1191 You must use : <i>LogFormat="%virtualname %host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"</i><br>
1192
1193 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
1194 <u>If your log records are EXACTLY like this (<b>NCSA combined and mod_gzip format 1</b> with <b>Apache 1.x</b>):</u><br>
1195 <i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 3904 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" mod_gzip: 66pct.</i><br>
1196 You must use : <i>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %other %gzipratio"</i><br>
1197 </td></tr></table>
1198 rizwank 1.1
1199 <u>If your log records are EXACTLY like this (<b>NCSA combined and mod_gzip format 2</b> with <b>Apache 1.x</b>):</u><br>
1200 <i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 3904 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" mod_gzip: DECHUNK:OK In:11393 Out:3904:66pct.</i><br>
1201 You must use : <i>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %other %other %gzipin %gzipout"</i><br>
1202
1203 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
1204 <u>If your log records are EXACTLY like this (<b>NCSA combined and mod_deflate</b> with <b>Apache 2 </b>):</u><br>
1205 <i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 3904 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" (45)</i><br>
1206 You must use : <i>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %deflateratio"</i><br>
1207 </td></tr></table>
1208
1209 <u>If your log records are EXACTLY like this (<b>NCSA combined with 2 spaces between some fields</b> with <b>Zope</b>):</u><br>
1210 <i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 3904 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" (45)</i><br>
1211 You must use : <i><br>
1212 LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"<br>
1213 LogSeparator=" *"<br>
1214 </i>
1215
1216 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
1217 <u>If your log records are EXACTLY like this (<b>NCSA common CLF</b> log format):</u><br>
1218 <i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 1234</i><br>
1219 rizwank 1.1 You must use : <i>LogFormat=4</i><br>
1220 Note: Browsers, OS's, Keywords and Referers features are not available with a such format.<br>
1221 </td></tr></table>
1222
1223 <u>If your log records are EXACTLY like this (With some <b>Squid</b> versions, after setting <i>emulate_http_log</i> to on):</u><br>
1224 <i>200.135.30.181 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET http://www.mydomain.com/page.html HTTP/1.0" 200 456 TCP_CLIENT_REFRESH_MISS:DIRECT</i><br>
1225 You must use : <i>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %other"</i><br>
1226
1227 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
1228 <u>If your log records are EXACTLY like this (Some old <b>IIS</b> W3C log format):</u><br>
1229 <i>yyyy-mm-dd hh:mm:ss 62.161.78.73 - GET /page.html 200 1234 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+NT+5.0) http://www.from.com/from.html</i><br>
1230 You must use : <i>LogFormat=2</i><br>
1231 </td></tr></table>
1232
1233 <u>If your log records are EXACTLY like this (Some <b>IIS</b> W3C log format with some <b>.net</b> servers):</u><br>
1234 <i>yyyy-mm-dd hh:mm:ss GET /page.html - 62.161.78.73 - Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+NT+5.0) http://www.from.com/from.html 200 1234 HTTP/1.1</i><br>
1235 You must use : <i>LogFormat="%time2 %method %url %logname %host %other %ua %referer %code %bytesd %other"</i><br>
1236
1237 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
1238 <u>If your log records are EXACTLY like this (Some <b>IIS 6+</b> W3C log format):</u><br>
1239 <i>yyyy-mm-dd hh:mm:ss GET /page.html - 62.161.78.73 - Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+NT+5.0) http://www.from.com/from.html 200 1234</i><br>
1240 rizwank 1.1 You must use : <i>LogFormat="date time cs-method cs-uri-stem cs-username c-ip cs-version cs(User-Agent) cs(Referer) sc-status sc-bytes"</i><br>
1241 </td></tr></table>
1242
1243 <u>If your log records are EXACTLY like this (With some <b>WebSite</b> versions):</u><br>
1244 <i>yyyy-mm-dd hh:mm:ss 62.161.78.73 - 192.168.1.1 80 GET /page.html - 200 11205 0 0 HTTP/1.1 mydomain.com Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+98) - http://www.from.com/from.html</i><br>
1245 You must use : <i>LogFormat="%time2 %host %logname %other %other %method %url %other %code %bytesd %other %other %other %other %ua %other %referer"</i><br>
1246
1247 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
1248 <u>If your log records are EXACTLY like this (<b>Webstar</b> native log format):</u><br>
1249 <i>05/21/00 00:17:31 OK 200 212.242.30.6 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt) http://www.cover.dk/ "www.cover.dk" :Documentation:graphics:starninelogo.white.gif 1133</i><br>
1250 You must use : <i>LogFormat=3</i><br>
1251 </td></tr></table>
1252
1253 <u>If your log records are EXACTLY like this (With some <b>Lotus Notes/Domino</b> versions):</u><br>
1254 <i>62.161.78.73 - Name Surname Service [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 1234 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"</i><br>
1255 You must use : <i>LogFormat=6</i><br>
1256
1257 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
1258 <u>If your log records are EXACTLY like this (<b>Lotus Notes/Domino 6.x</b> log format):</u><br>
1259 <i>62.161.78.73 - "Name Surname" Service [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 1234 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"</i><br>
1260 You must use : <i>LogFormat="%host %other %lognamequot %time1 %methodurl %code %bytesd %refererquot %uaquot"</i><br>
1261 rizwank 1.1 </td></tr></table>
1262
1263 <u>If your log records are EXACTLY like this (With <b>Oracle9iAS</b>):</u><br>
1264 <i>62.161.78.73 - [dd/mmm/yyyy:hh:mm:ss +0x00] GET /page.html HTTP/1.1 200 1234 - "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"</i><br>
1265 Where separators are <tab> chars or several spaces,
1266 You must use : <i>LogFormat="%host %logname %time1 %method %url %other %code %bytesd %referer %uaquot"</i>
1267 and <i>LogSeparator="\s+"</i><br>
1268
1269 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
1270 <u>If you use a FTP server like <b>ProFTPd</b>:</u><br>
1271 See <a href="#FTP">FAQ-COM090</a>.<br>
1272 </td></tr></table>
1273
1274 <u>If you want to analyze a mail log file (<b>Postfix</b>, <b>Sendmail</b>, <b>QMail</b>, <b>MDaemon</b>, <b>Exchange</b>):</u><br>
1275 See <a href="#MAIL">FAQ-COM100</a>.<br>
1276
1277 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
1278 <u>If you use a Media Server (<b>Realmedia</b>, <b>Windows Media Server</b>):</u><br>
1279 See <a href="#MEDIASERVER">FAQ-COM110</a>.<br>
1280 </td></tr></table>
1281
1282 rizwank 1.1 <u>If your log records are EXACTLY like this (With some providers):</u><br>
1283 <i>62.161.78.73 - - [dd/Month/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" "-" 200 1234</i><br>
1284 You must use : <i>LogFormat="%host %other %logname %time1 %methodurl %other %code %bytesd"</i><br>
1285 Note: Browsers, OS's, Keywords and Referers features are not available with a such format.<br>
1286
1287 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
1288 <u>There is a lot of other possible log formats.</u><br>
1289 You must use a personalized log format LogFormat ="..." as described in config file to
1290 support other various log formats.<br>
1291 </td></tr></table>
1292 <br>
1293
1294
1295 <a name="CORRUPTEDDROPPED"></a><br>
1296 <b><u>FAQ-SET270 : ONLY CORRUPTED OR DROPPED RECORDS</u></b><br>
1297 <font class=CProblem>PROBLEM:</font><br>
1298 After running an AWStats update process, all my records are reported to be corrupted or dropped<br>
1299 <font class=CSolution>SOLUTION:</font><br>
1300 First, if you have only a small percent of corrupted or dropped records, don't worry. This
1301 is a normal behaviour. Few corrupted or dropped records can appear in a log file because of
1302 internal web server bug, virus attack, error writing, log purge or rotate during a writing, etc...<br>
1303 rizwank 1.1 However, if ALL your records are reported to be corrupted or dropped, check the following things:<br>
1304 If they are all dropped, run the update process from command line adding the option -showdropped<br>
1305 -> You will be able to know why a dropped record is discarded. In most cases, this is because you use a
1306 too large or bad filter parameter (SkipFiles, SkipHosts, OnlyFiles ...).<br>
1307 If they are all corrupted, run the update process from command line adding the option -showcorrupted<br>
1308 -> You will be able to know why a corrupted record is discarded.<br>
1309 If this is because of the log format, check the <a href="#EMPTY_STATS">FAQ-SET350</a> about log format errors.<br>
1310 If this is because the date of a record is said to be lower than date of previous, this means
1311 that you ran update processes on different log files without keeping the chronological order
1312 of log files.<br>
1313 If this is because the date is invalid, you might have a problem of date not computed correctly
1314 this it happens in some Pentium4/Xeon4 processors:<br>
1315 On some (few) Intel Pentium4 (also Xeon4) based host systems, log file time can not be computed
1316 correctly. This is not an issue of AWStats itself. This error usually occurs on source-based linux
1317 distributions (gentoo, slackware etc.), where all system libraries are compiled with CPU optimization.
1318 AWStats is a highly developed PERL application. PERL itself relies on some system libraries,
1319 for example GLIBC. The GLIBC library usually is buggy in this case. There is an easy way to figure
1320 out whether the problem described here is responsible for AWStats problems on your system:<br>
1321 If you have shell access to your machine, simply type the following command:<br>
1322 <i>perl -e "print int ('541234567891011165415658')"</i><br>
1323 (NOTE: any 25-digit number works, there is no need to type this exact number)<br>
1324 rizwank 1.1 If everything goes fine, you should see a floating point number as output:<br>
1325 <i>5.41234567891011e+23</i><br>
1326 In this case, please do more research on your log file formats. Your host system itself is not
1327 responsible for the error.<br>
1328 But if simply a "0" returns or some other error, this is an indication of your glibc beeing corrupt.<br>
1329 ATTENTION: The only solution in this case might be to recompile your GLIBC. This can be a quite tricky
1330 task. Please consult the documentation and F.A.Q.s of your linux distribution first!! (experts: first
1331 check your global compile flags, eg. march=Pentium4. Trying with other compile flags can solve problem
1332 quickly in some cases.<br>
1333 NOTE: In some cases, this error might occur "suddenly", even though AWStats was already running
1334 perfect already.<br>
1335 <br>
1336
1337 <a name="NOTSAMENUMBER"></a><br>
1338 <b><u>FAQ-SET280 : ERROR "NOT SAME NUMBER OF RECORDS OF..."</u></b><br>
1339 <font class=CProblem>PROBLEM:</font><br>
1340 When I run AWStats from command line (or as a cgi from a browser), I get
1341 a message "Not same number of records of ...".<br>
1342 <font class=CSolution>SOLUTION:</font><br>
1343 This means your AWStats reference database files (operating systems, browsers, robots...) are not correct.
1344 First try to update to last version. Then check in your disk that you have only ONE of those files. They should be
1345 rizwank 1.1 in '<b>lib</b>' directory ('db' with 4.0) where awstats.pl is installed:<br>
1346 <i>browsers.pm</i><br>
1347 <i>domains.pm</i><br>
1348 <i>operating_systems.pm</i><br>
1349 <i>robots.pm</i><br>
1350 <i>search_engines.pm</i><br>
1351 <i>worms.pm</i><br>
1352 <i>status_http.pm</i><br>
1353 <i>status_smtp.pm</i><br>
1354 <br>
1355
1356 <a name="COULDNOTOPEN"></a><br>
1357 <b><u>FAQ-SET300 : ERROR "COULDN'T OPEN FILE ..."</u></b><br>
1358 <font class=CProblem>PROBLEM:</font><br>
1359 I have the following error:<br>
1360 <i>"Couldn't open file /workingpath/awstatsmmyyyy.tmp.9999: Permission denied."</i><br>
1361 <font class=CSolution>SOLUTION:</font><br>
1362 This error means that the web server didn't succeed in writing the working temporary file (file ended by .tmp.9999
1363 where 9999 is a number) because of permissions problems.<br>
1364 First check that the directory <i>/workingpath</i> has "Write" permission for<br>
1365 user <u>nobody</u> (default user used by Apache on Linux systems)<br>
1366 rizwank 1.1 or user <u>IUSR_<i>SERVERNAME</i></u> (default used user by IIS on NT).<br>
1367 With Unix, try with a path with no links.<br>
1368 With NT, you must check NTFS permissions ("Read/Write/Modify"), if your directory is on a NTFS partition.<br>
1369 With IIS, there is also a "Write" permission attribute, defined in directory properties
1370 in your IIS setup, that you must check.<br>
1371 With IIS, if a default cgi-bin directory was created during IIS install, try to
1372 put AWStats directly into this directory.<br>
1373 If this still fails, you can change the DirData parameter to say AWStats
1374 that you want to use another directory (A directory you are sure that the default
1375 user, used by web server process, can write into).<br>
1376 <br>
1377
1378 <a name="MALFORMEDUTF8"></a><br>
1379 <b><u>FAQ-SET320 : ERROR "MALFORMED UTF-8 CHARACTER (UNEXPECTED ..."</u></b><br>
1380 <font class=CProblem>PROBLEM:</font><br>
1381 When running AWStats from command line, I get one or several lines like this
1382 on my output:<br>
1383 <i>Malformed UTF-8 character (unexpected non-continuation byte 0x6d, immediately after start byte 0xe4) at /www/cgi-bin/lib/xxx.pm line 999.</i>
1384 <br>
1385 <font class=CSolution>SOLUTION:</font><br>
1386 This problem appeared with RedHat 8 and Perl 5.8.<br>
1387 rizwank 1.1 I don't know if RedHat provides a fix for this, but some users had reported that you can
1388 remove thoose warmless messages by changing your LANG environment variable, removing
1389 the ".UTF-8" at the end. For example, set <i>LANG="en_US"</i> instead of <i>LANG="en_US.UTF8"</i><br>
1390 <br>
1391
1392 <a name="EMPTY_STATS"></a><br>
1393 <b><u>FAQ-SET350 : EMPTY OR NULL STATISTICS REPORTED</u></b><br>
1394 <font class=CProblem>PROBLEM:</font><br>
1395 AWStats seems to work but I'm not getting any results. i get a statistics page that looks like i have no hits.<br>
1396 <font class=CSolution>SOLUTION:</font><br>
1397 That's one of the most common problem you can get and there is 3 possible reasons :<br>
1398 <br>
1399 1) Your log file format setup might be wrong.<br>
1400 <u>If you use Apache web server</u><br>
1401 The best way of working is to use the <i>"combined"</i> log format (See the <a href="awstats_setup.html">Setup and Use</a> page
1402 to know the way to change your Apache server log from <i>"common"</i> log format into <i>"combined"</i>).
1403 Don't forget to stop Apache, reset your log file and restart Apache to make change into combined
1404 effective. Then you must setup your AWStats config file with value <a href="awstats_config.html#LogFormat">LogFormat</a>=1.<br>
1405 If you want to use another format, read the next FAQ to have examples of LogFile value according
1406 to log files format.<br>
1407 <u>If you use IIS server or Windows built-in web server</u><br>
1408 rizwank 1.1 The Internet Information Server default W3C Extended Log Format will not work correctly with AWStats.
1409 To make it work correctly, start the IIS Snap-in, select the web site and look at it's Properties.
1410 Choose W3C Extended Log Format, then Properties, then the Tab Extended Properties and uncheck everything
1411 under Extended Properties. Once they are all unchecked, check off the list given in
1412 the <a href="awstats_setup.html">Setup and Use</a> page ("With IIS Server" chapter).<br>
1413 You can also read the next FAQ to have examples of <a href="awstats_config.html#LogFormat">LogFormat</a> value according to log files format.<br>
1414 <br>
1415 2) You are viewing stats for a year or month when no hits was made on your server.<br>
1416 When you run awstats, the reports is by default for the current month/year.<br>
1417 If you want to see data for another month/year you must:<br>
1418 Add -year=YYYY -month=MM on command line when building the html report page from command line.<br>
1419 Use an URL like http://myserver/cgi-bin/awstats.pl?config=xxx&year=YYYY&month=MM if viewing stats with AWStats used as a CGI.<br>
1420 <br>
1421 3) When you read your statistics, AWStats does not use the same config file than the one
1422 used for the update process. Scan your disk for files that match <i>awstats.*conf</i> and remove
1423 all files that are not the config file(s) you need (awstats.conf files, if found, can be deleted.
1424 It is better to use a config file called awstats.mydomain.conf).<br>
1425 <br>
1426
1427 <a name="PARTIAL_STATS"></a><br>
1428 <b><u>FAQ-SET360 : STATISTICS REPORTED EXCEPT FOR OS, BROWSERS, ROBOTS AND KEYWORDS/KEYPHRASES</u></b><br>
1429 rizwank 1.1 <font class=CProblem>PROBLEM:</font><br>
1430 AWStats seems to report my statistics however some charts, like robots, os', browsers, search engines,
1431 or keywords/keyphrases are empty.<br>
1432 <font class=CSolution>SOLUTION:</font><br>
1433 If only robots, search engines or keywords/keyphrases are empty, this simply means your web site was not
1434 yet visited by any robots and noone found your site using a search engines (this happens particularly
1435 for Intranet which are not referenced on search engines).
1436 If all of them are empty or with only unknown values, even after several updates, this probably means
1437 that your logfile does not contains all informations, this happens with Apache when using the
1438 standard <i>"common"</i> log format instead of the standard <i>"combined"</i> log format.<br>
1439 You may also use LogFormat=4 into your AWStats config files instead of 1.<br>
1440 Read AWStats setup documentation to known how to setup your Apache Web server to report logs in
1441 a "combined" log format then set <a href="awstats_config.html#LogFormat">LogFormat</a>=1 into your
1442 AWStats config file.<br>
1443 <br>
1444
1445 <a name="REDIRECT"></a><br>
1446 <b><u>FAQ-SET400 : PIPE REDIRECTION TO A FILE GIVE ME AN EMPTY FILE</u></b><br>
1447 <font class=CProblem>PROBLEM:</font><br>
1448 I want to redirect awstats.pl output to a file with the following command :<br>
1449 <i>> awstats.pl -config=... [other_options] > myfile.html</i><br>
1450 rizwank 1.1 But myfile.html is empty (size is 0). If i remove the redirection, everythings works correctly.<br>
1451 <font class=CSolution>SOLUTION:</font><br>
1452 This is not an AWStats bug but a problem between perl and Windows.<br>
1453 You can easily solve this running the following command instead:<br>
1454 <i>> perl awstats.pl -config=... [other_options] > myfile.html</i><br>
1455 <br>
1456
1457 <a name="NO_ICON"></a><br>
1458 <b><u>FAQ-SET450 : NO PICTURES/GRAPHICS SHOWN</u></b><br>
1459 <font class=CProblem>PROBLEM:</font><br>
1460 AWStats seems to work (all data and counters seem to be good) but I have no image shown.<br>
1461 <font class=CSolution>SOLUTION:</font><br>
1462 With Apache web server, you might have troubles (no picture shown on stats page) if you use a directory called "icons" (because of Apache
1463 pre-defined "icons" alias directory), so use instead, for example, a directory called "icon" with
1464 no s at the end (Rename your directory physically and change the <a href="awstats_config.html#DirIcons">DirIcons</a> parameter in config file
1465 to reflect this change).<br>
1466 <br>
1467
1468 <a name="MIGRATEDOUBLED"></a><br>
1469 <b><u>FAQ-SET700 : MY VISITS ARE DOUBLED FOR OLD MONTH I MIGRATED FROM 3.2 TO 5.X</u></b><br>
1470 <font class=CProblem>PROBLEM:</font><br>
1471 rizwank 1.1 After having migrated an old history file for a month, the number of visits for this month is doubled.
1472 So the number of "visits per visitor" is also doubled and "pages per visit" and "hits per visit" is divided by 2.
1473 All other data like "pages", "hits" and bandwith are correct.<br>
1474 <font class=CSolution>SOLUTION:</font><br>
1475 This problem occurs when migrating history files from 3.2 to 5.x.<br>
1476 To fix this you can use the following tip (warning, do this only after migrating from
1477 3.2 to 5.x and if your visit value is doubled). The goal is to remove the line
1478 in history file that looks like this<br>
1479 <i>YYYYMM00 999 999 999 999</i><br>
1480 where YYYY and MM are year and month of config file and 999 are numerical values.<br>
1481 <br>
1482 <u>So if your OS is Unix/Linux</u><br>
1483 <i>
1484 grep -vE '^[0-9]{6}00' oldhistoryfile > newhistoryfile<br>
1485 mv newhistoryfile oldhistoryfile<br>
1486 </i>
1487 And then run the migrate process again on the file.<br>
1488 <br>
1489 <u>If your OS is windows and got cygwin</u><br>
1490 You must follow same instructions than if OS is Unix/Linux BUT
1491 you must do this from a cygwin 'sh' shell and not from the DOS
1492 rizwank 1.1 prompt (because the ^ is not understanded by DOS).<br>
1493 And then run the migrate process again on the file.<br>
1494 <br>
1495 <u>In any other case (in fact works for every OS)</u><br>
1496 You must remove manually the line <i>YYYYMM00 999 999 999 999</i> (must find one and only
1497 one such line) and then run the migrate process again on the file.<br>
1498 <br>
1499
1500 <a name="OUTOFMEMORYCYGWIN"></a><br>
1501 <b><u>FAQ-SET750 : AWSTATS RUN OUT OF MEMORY DURING UPDATE PROCESS WITH CYGWIN PERL</u></b><br>
1502 <font class=CProblem>PROBLEM:</font><br>
1503 When I run the update process on a large log file with cygwin Perl, AWStats run out of memory
1504 but I am sure that I have enough memory to run AWStats according to the 'memory' column in
1505 benchmark chart available in AWStats documentation (<a href="awstats_benchmark.html">benchmark page</a>).<br>
1506 <font class=CSolution>SOLUTION:</font><br>
1507 It might be a limit inside Cygwin Perl.
1508 Try to increase the Cygwin parameter <i>heap_chunk_in_mb</i>.<br>
1509 <br>
1510
1511 <a name="SPEED"></a><br>
1512 <b><u>FAQ-SET800 : AWSTATS SPEED/TIMEOUT PROBLEMS ?</u></b><br>
1513 rizwank 1.1 <font class=CProblem>PROBLEM:</font><br>
1514 When I analyze large log files, processing times are very important (Example: update process from a browser
1515 returns a timeout/internal error after a long wait).
1516 Is there a setup or things to do to avoid this and increase speed ?<br>
1517 <font class=CSolution>SOLUTION:</font><br>
1518 You really need to understand how a log analyzer works to have good speed.
1519 There is also major setup changes you can do to decrease your processing time.<br>
1520 See <a href="awstats_benchmark.html#ADVICES">important advices</a> in benchmark page.<br>
1521 <br>
1522
1523
1524 <hr><br>
1525
1526
1527 <a name="CSSATTACK"></a><br>
1528 <b><u>FAQ-SEC100 : CAN AWSTATS BE USED TO MAKE CROSS SITE SCRIPTING ATTACKS ?</u></b><br>
1529 <font class=CProblem>PROBLEM:</font><br>
1530 If a bad user use a browser to make a hit on an URL that include a < SCRIPT > ... < /SCRIPT >
1531 section in its parameter, when AWStats will show the links on the report page, does the script will be executed ?<br>
1532 <font class=CSolution>SOLUTION:</font><br>
1533 No. AWStats use a filter to remove all scripts codes that was included in an URL to make a Cross Site Scripting Attack using a
1534 rizwank 1.1 log analyzer report page.<br>
1535 <br>
1536
1537 <a name="SECUSER"></a><br>
1538 <b><u>FAQ-SEC150 : HOW CAN I PREVENT SOME USERS TO SEE STATISTICS OF OTHER USERS ?</u></b><br>
1539 <font class=CProblem>PROBLEM:</font><br>
1540 I don't want a user xxx (having a site www.xxx.com) to see statistics of user yyy (having
1541 a site www.yyy.com). How can i setup AWStats for this ?<br>
1542 <font class=CSolution>SOLUTION:</font><br>
1543 Take a look at the <a href="awstats_security.html">security page</a>.<br>
1544 <br>
1545
1546 <a name="WORMS"></a><br>
1547 <b><u>FAQ-SEC200 : HOW TO MANAGE LOG FILES (AND STATISTICS) CORRUPTED BY 'WORMS' ATTACKS ?</u></b><br>
1548 <font class=CProblem>PROBLEM:</font><br>
1549 My site is attacked by some worms viruses (like Nimba, Code Red...). This make my log file corrupted
1550 and full of 404 errors. So my statistics are also full of 404 errors. This make AWStats slower and my history files very large.
1551 Can I do something to avoid this ?<br>
1552 <font class=CSolution>SOLUTION:</font><br>
1553 Yes.<br>
1554 'Worms' attacks are infected browsers, robots or server changed into web client that make hits on your site using a very long
1555 rizwank 1.1 unknown URL like this one:<br>
1556 <i>/default.ida?XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%40%50...%40%50</i><br>
1557 URL is generated by the infected robot and the purpose is to exploit a vulnerability of the web server (In most cases, only IIS is vulnerable).
1558 With such attacks, you will will always find a 'common string' in those URLs.
1559 For example, with Code Red worm, there is always default.ida in the URL string. Some other worms send URLs with cmd.exe in it.<br>
1560 With 6.0 version and higher, you can set the <a href="awstats_config.html#LevelFor">LevelForWormsDetection</a>
1561 parameter to "2" and <a href="awstats_config.html#Show">ShowWormsStats</a> to "HBL" in
1562 config file to enable the worm filtering nd reporting.<br>
1563 However, this feature reduce seriously AWStats speed and the worms database (lib/worms.pm file) can't contain
1564 all worms signatures. So if you still have rubish hits, you can modify the worms.pm file yourself or
1565 edit your config file to add in the <a href="awstats_config.html#SkipFiles">SkipFiles</a> parameter some
1566 values to discard the not required records, using a regex syntax like example :<br>
1567 <i>SkipFiles="REGEX[^\/default\.ida] REGEX[\/winnt\/system32\/cmd\.exe]"</i><br>
1568 <br>
1569
1570 <hr>
1571
1572 <script language=javascript>
1573 var date='$Date: 2005/01/15 23:30:52 $';
1574 document.writeln("Last revision: "+date);
1575 </script>
1576 rizwank 1.1
1577 </body>
1578 </html>
1579
|
Return to
awstats_faq.html
CVS log
Up to [RizwankCVS] / geekymedia_web / awstats-6.3 / docs