Scripts Archives - C:Amie (not) Com!

System Requirements:

Windows NT 4.0 Workstation
Windows 2000 Professional
Windows XP
Windows NT 4.0 Server
Windows 2000 Server
Windows Server 2003, R2
Windows Server 2008, R2
Windows Server 2012
Windows Server 2016
Windows Server 2019
Internet Information Services 4.0, 5.0, 5.1, 6.0, 7.0, 7.5, 8.0, 10
Analog CE

The Problem:

This article discusses how to automate the dissemination of statistics data to multiple web site clients on a typical, single instance production IIS web server. The article discusses the configuration of Analog and the file system considerations required to automate the delivery of statistics to all IIS web sites hosted on the server instance.

More Info

In the default Analog CE sample config file, Analog is configured to support a single web site through a single instance of the Analog executive.

In a larger IIS web host environment, a single IIS server may potentially host hundreds or even thousands of individual web sites. In this model, it is necessary to tweak the configuration of Analog to act as a single instance, multi-target model – instead of the default single instance, single target model.

How-to

The how-to is split into the following sections:

What is not covered here
Pre-requisites
File System Considerations
Prepare Analog
Configure the Analog Server Settings
Configure the Analog Site Settings
Running Analog on a Schedule

What is not covered here

This guide is specifically written to demonstrate the configuration of Analog. To that end it does not cover the installation, setup or configuration of IIS in either a manual or an automatic capacity.

This guide will provide you with a scalable solution towards running Analog on a production IIS server, however most administrators will wish to integrate this process into the creation of web sites on IIS.

Pre-requisites

You will need to ensure that scripts run through an account with appropriate permissions to be able to write to the output folders on your web server. If you are using UAC on your server, ensure that you execute the scripts with elevation.

File System Considerations

By default, IIS provides a separation between the sites web root and the logs associated with that web site instance. By default IIS bases its web site root under c:\inetpub and the log files necessary for Analog to function are stored under c:\windows\system32\LogFiles\<Instance>.

Hosting providers should move these files to a different location, one that is preferably separate from the operating system volume. In most cases, these will be accessible by the tenant. How you wish to do this will depend primarily on whether you want your client users to gain access to the logs or not.

A suggestion for how to structure your web sites on disk in both scenarios is shown below. In both examples the client would FTP directly into d:\sites\<domain> to upload or download content.

User has access to Logs	User does not have access to Logs
`d:` `- \sites` `- \www.domain.com` `- \web` `- \stats` `- \logs` `- \W3SVC1` `- \www.website.net` `- \web` `- \stats` `- \logs` `- \W3SVC2`	`d:` `- \sites` `- \www.domain.com` `- \stats` `- \www.website.net` `- \stats` `- \logs` `- \W3SVC1` `- \W3SVC2`
In this example the client has access to their own log files (presumably over SFTP). This allows them to download their logs for offline analysis, however you will also generate an Analog report for them into the \stats folder under their web site root folder (\web).	In this example, log files are available for internal users only, you plan to use these to generate a report for the client using Analog into the \stats folder.

Prepare Analog

To fit in with the file-system structure outlined above, this guide will assume that the following file system structure is in use:

d:
- \parser
-   \Analog
-   \ReportMagic
- \sites
-   \www.domain.com
-      \web
-        \stats
-      \logs
-        \W3SVC1
-   \www.website.net
-      \web
-        \stats
-      \logs
-        \W3SVC2

Copy the contents of the latest Analog CE binary zip file into d:\parser\Analog.

Configure the Analog Server Settings

Analog’s global analog.cfg file must be reconfigured to be site independent so that it becomes the global configuration file. This is achieved by removing all references to a particular web site to create a baseline configuration. In essence, you are creating the default values for all Analog executions that will be used unless overridden in the local config.

Open the analog.cfg file in notepad.

Using the Analog CE Configuration File Generator output as an example, you should remove the highlighted lines to form a baseline configuration.

# Analog CE Baseline Statistics Configuration

# Version 1.0.1

# See http://www.c-amie.co.uk/ for updates and more
LOGFILE c:\windows\system32\logfiles\w3svc1\*.logOUTPUT HTML

OUTFILE c:\inetpub\wwwroot\stats\index.html

HOSTNAME "Analog Test Site"

HOSTURL http://www.domain.com/

IMAGEDIR "images/"

STYLESHEET images/analog.css

# Reports Enabled/Disabled List
ALL ON
ALLCHART ON GENERAL ON #General Summary
YEARLY ON #Yearly Report
QUARTERLY ON #Quarterly Report
MONTHLY ON #Monthly Report
WEEKLY ON #Weekly Report
DAILYREP ON #Daily Report
DAILYSUM ON #Daily Summary
HOURLYREP ON #Hourly Report
HOURLYSUM ON #Hourly Summary
WEEKHOUR ON #Hour of the Week Summary
QUARTERREP ON #Quarter-Hour Report
QUARTERSUM ON #Quarter-Hour Summary
FIVEREP ON #Five-Minute Report
FIVESUM ON #Five-Minute Summary
HOST ON #Host Report
REDIRHOST ON #Host Redirection Report
FAILHOST ON #Host Failure Report
ORGANISATION ON #Organisation Report
DOMAIN ON #Domain Report
REQUEST ON #Request Report
DIRECTORY ON #Directory Report
FILETYPE ON #File Type Report
SIZE ON #File Size Report
PROCTIME ON #Processing Time Report
REDIR ON #Redirection Report
FAILURE ON #Failure Report
REFERRER ON #Referrer Report
REFSITE ON #Referring Site Report
SEARCHQUERY ON #Search Query Report
SEARCHWORD ON #Search Word Report
INTSEARCHQUERY ON #Internal Search Query Report
INTSEARCHWORD ON #Internal Search Word Report
REDIRREF ON #Redirected Referrer Report
FAILREF ON #Failed Referrer Report
BROWSERREP ON #Browser Report
BROWSERSUM ON #Browser Summary
OSREP ON #Operating System Report
VHOST ON #Virtual Host Report
REDIRVHOST ON #Virtual Host Redirection Report
FAILVHOST ON #Virtual Host Failure Report
USER ON #User Report
REDIRUSER ON #User Redirection Report
FAILUSER ON #User Failure Report
STATUS ON #Status Code Report

# Referring URL Report
REFLINKINCLUDE *
REFREPEXCLUDE http://domain.com/*
REFREPEXCLUDE http://www.domain.com/*
REFFLOOR 1r

# Referring Site
REFSITEEXCLUDE http://domain.com/
REFSITEEXCLUDE http://www.domain.com/

# Request Report
REQFLOOR 1r
REQEXCLUDE *.jpg
REQEXCLUDE *.gif
REQEXCLUDE *.png
REQEXCLUDE *.bmp
REQEXCLUDE *.class
REQEXCLUDE *.js

# Status Code Report
304ISSUCCESS ON # Includes 304 errors on the request report

# Redirected Referrers Report
REDIRREFLINKINCLUDE *

# Failed Referrers Report
FAILREFLINKINCLUDE *

# Browser Summary
SUBBROW */*

# Operating System Report
OSCHARTEXPAND Windows

# Default Documents
# PAGEINCLUDE *.shtml
PAGEINCLUDE *.asp
PAGEINCLUDE *.aspx
# PAGEINCLUDE *.jsp
# PAGEINCLUDE *.cfm
# PAGEINCLUDE *.pl
# PAGEINCLUDE *.php
# PAGEINCLUDE *.rb

# Custom Exclusions
#FILEEXCLUDE /admin/*

# Robots & Crawlers
ROBOTINCLUDE REGEXPI:robot
ROBOTINCLUDE REGEXPI:spider
ROBOTINCLUDE REGEXPI:crawler ROBOTINCLUDE Baiduspider/*
ROBOTINCLUDE bingbot/*
ROBOTINCLUDE Googlebot*
ROBOTINCLUDE Infoseek*
ROBOTINCLUDE msnbot*
ROBOTINCLUDE Scooter*
ROBOTINCLUDE *Slurp*
ROBOTINCLUDE Ultraseek*
ROBOTINCLUDE *Validator*
ROBOTINCLUDE YandexBot/*
ROBOTINCLUDE YodaoBot/* # Search Engines
SEARCHENGINE http://*altavista.*/* q
SEARCHENGINE http://*yahoo.*/* p
SEARCHENGINE http://*google.*/* q,as_q,as_epq,as_oq
SEARCHENGINE http://*bing.*/* q
SEARCHENGINE http://*lycos.*/* query
SEARCHENGINE http://*aol.*/* query
SEARCHENGINE http://*excite.*/* search
SEARCHENGINE http://*go2net.*/* general
SEARCHENGINE http://*metacrawler.*/* general
SEARCHENGINE http://*msn.*/* MT
SEARCHENGINE http://*hotbot.com/* MT
SEARCHENGINE http://*netscape.*/* search
SEARCHENGINE http://*looksmart.*/* key
SEARCHENGINE http://*infoseek.*/* qt
SEARCHENGINE http://*webcrawler.*/* search,searchText
SEARCHENGINE http://*goto.*/* Keywords
SEARCHENGINE http://*snap.*/* keyword
SEARCHENGINE http://*dogpile.*/* q
SEARCHENGINE http://*bbc.*/* q
SEARCHENGINE http://*askjeeves.*/* ask
SEARCHENGINE http://*ask.*/* ask
SEARCHENGINE http://*aj.*/* ask
SEARCHENGINE http://*directhit.*/* qry
SEARCHENGINE http://*alltheweb.*/* query
SEARCHENGINE http://*naver.*/* query
SEARCHENGINE http://*northernlight.*/* qr
SEARCHENGINE http://*nlsearch.*/* qr
SEARCHENGINE http://*dmoz.*/* search
SEARCHENGINE http://*newhoo.*/* search
SEARCHENGINE http://*netfind.*/* query,search,s
SEARCHENGINE http://*/netfind* query
SEARCHENGINE http://*/pursuit query
SEARCHENGINE http://*/mamma.*/* query
SEARCHENGINE http://*ixquick.*/* metasearch.pl
SEARCHENGINE http://*vivisimo.*/* search
SEARCHENGINE http://*mysearch.*/* searchfor

# Static Internet Documents
TYPEOUTPUTALIAS .html ".html [Hypertext Markup Language]"
TYPEOUTPUTALIAS .htm ".htm [Hypertext Markup Language]"
TYPEOUTPUTALIAS .shtml ".shtml [Server-parsed HTML]"
TYPEOUTPUTALIAS .ps ".ps [PostScript]"
TYPEOUTPUTALIAS .gz ".gz [Gzip compressed files]"
TYPEOUTPUTALIAS .tar.gz ".tar.gz [Compressed archives]"
TYPEOUTPUTALIAS .txt ".txt [Plain text Documents]"
TYPEOUTPUTALIAS .cdf ".cdf [Channel Definition File]"

# Scripting & Dynamic Internet Content Files
TYPEOUTPUTALIAS .asp ".asp [Active Server Pages]"
TYPEOUTPUTALIAS .aspx ".aspx [Active Server Pages .net]"
TYPEOUTPUTALIAS .cgi ".cgi [CGI scripts]"
TYPEOUTPUTALIAS .pl ".pl [Perl scripts]"
TYPEOUTPUTALIAS .css ".css [Cascading Style Sheets]"
TYPEOUTPUTALIAS .class ".class [Java class files]"
TYPEOUTPUTALIAS .hqx ".hqx [Macintosh archives]"
TYPEOUTPUTALIAS .jsp ".jsp [Java Server Pages]"
TYPEOUTPUTALIAS .cfm ".cfm [Cold Fusion]"
TYPEOUTPUTALIAS .php ".php [PHP Hypertext Processor]"
TYPEOUTPUTALIAS .js ".js [JavaScript code]"
TYPEOUTPUTALIAS .dll ".dll [Dynamic Link Library]"
TYPEOUTPUTALIAS .asa ".asa [Web Server Scripting Configuration]"
TYPEOUTPUTALIAS .url ".url [Windows Internet Shortcut]"
TYPEOUTPUTALIAS .lnk ".lnk [Windows Explorer Shortcut]"
TYPEOUTPUTALIAS .ini ".ini [Configuration Settings File]"
TYPEOUTPUTALIAS .log ".log [Log Files]"
TYPEOUTPUTALIAS .diz ".diz [DIZ Text File]"
TYPEOUTPUTALIAS .inc ".inc [SSI Inclusion File]"
TYPEOUTPUTALIAS .xml ".xml [eXtensible Markup Language File]"
TYPEOUTPUTALIAS .rdf ".rdf [Resource Description Framework File]"
TYPEOUTPUTALIAS .rb ".rb [Ruby script file]"

# Image Files
TYPEOUTPUTALIAS .jpg ".jpg [JPEG graphics]"
TYPEOUTPUTALIAS .jpeg ".jpeg [JPEG graphics]"
TYPEOUTPUTALIAS .jpe ".jpe [JPEG graphics]"
TYPEOUTPUTALIAS .gif ".gif [GIF graphics]"
TYPEOUTPUTALIAS .gfa ".gfa [GIF graphics]"
TYPEOUTPUTALIAS .png ".png [Portable Network Graphics]"
TYPEOUTPUTALIAS .bmp ".bmp [BitMap]"
TYPEOUTPUTALIAS .bmz ".bmz [BitMap]"
TYPEOUTPUTALIAS .dib ".dib [BitMap]"
TYPEOUTPUTALIAS .rle ".rle [BitMap]"
TYPEOUTPUTALIAS .2bp ".2bp [Windows CE 4 Tone BitMap]"
TYPEOUTPUTALIAS .ico ".ico [Icon File]"
TYPEOUTPUTALIAS .tif ".tif [Tag Image File Format]"
TYPEOUTPUTALIAS .tiff ".tiff [Tag Image File Format]"
TYPEOUTPUTALIAS .wmf ".wmf [Windows Metafile (ClipArt)]"
TYPEOUTPUTALIAS .pct ".pct [Macintosh PICT]"
TYPEOUTPUTALIAS .pict ".pict [Macintosh PICT]"
TYPEOUTPUTALIAS .pcz ".pcz [Macintosh PICT Compressed]"
TYPEOUTPUTALIAS .pcd ".pcd [Kodak Photo CD]"
TYPEOUTPUTALIAS .pcx ".pcx [PC Paintbrush]"
TYPEOUTPUTALIAS .cdr ".cdr [Corel Draw]"
TYPEOUTPUTALIAS .cgm ".cgm [COmputer Graphics Metafile]"
TYPEOUTPUTALIAS .eps ".eps [Encapsulated PostScript]"
TYPEOUTPUTALIAS .fpx ".fpx [FPX Format]"
TYPEOUTPUTALIAS .wpg ".wpg [WordPerfect Graphics]"
TYPEOUTPUTALIAS .mix ".mix [Picture IT! Format]"
TYPEOUTPUTALIAS .psd ".psd [Adobe Photoshop Document]"

# Multimedia Audio, Video & Misc
TYPEOUTPUTALIAS .wav ".wav [WAV sound files]"
TYPEOUTPUTALIAS .avi ".avi [AVI movies]"
TYPEOUTPUTALIAS .arc ".arc [Compressed archives]"
TYPEOUTPUTALIAS .mid ".mid [MIDI sound files]"
TYPEOUTPUTALIAS .midi ".midi [MIDI sound files]"
TYPEOUTPUTALIAS .rmi ".rmi [MIDI sound files]"
TYPEOUTPUTALIAS .ivf ".ivf [Indeo Video Format movie]"
TYPEOUTPUTALIAS .aif ".aif [AIFF sound files]"
TYPEOUTPUTALIAS .aifc ".aifc [AIFF sound files]"
TYPEOUTPUTALIAS .aiff ".aiff [AIFF sound files]"
TYPEOUTPUTALIAS .au ".au [AU sound files]"
TYPEOUTPUTALIAS .snd ".snd [AU sound files]"
TYPEOUTPUTALIAS .mp3 ".mp3 [MP3 sound files]"
TYPEOUTPUTALIAS .m4a ".m4a [MPEG 4 Audio]"
TYPEOUTPUTALIAS .m4v ".m4v [MPEG 4 Video]"
TYPEOUTPUTALIAS .mov ".mov [Quick Time movie]"
TYPEOUTPUTALIAS .mpg ".mpg [MPEG movie]"
TYPEOUTPUTALIAS .mpeg ".mpeg [MPEG movie]"
TYPEOUTPUTALIAS .m1v ".m1v [MPEG 1 Video]"
TYPEOUTPUTALIAS .mp2v ".mp2v [MPEG 2 Video]"
TYPEOUTPUTALIAS .mpe ".mpe [MPEG movie]"
TYPEOUTPUTALIAS .wax ".wax [Windows Media Audio Extension]"
TYPEOUTPUTALIAS .wvx ".wvx [Windows Media Video Extension]"
TYPEOUTPUTALIAS .m3u ".m3u [MPEG 3 Audio]"
TYPEOUTPUTALIAS .wma ".wma [Windows Media Audio]"
TYPEOUTPUTALIAS .wmv ".wmv [Windows Media Video]"
TYPEOUTPUTALIAS .ra ".ra [Real Audio File]"
TYPEOUTPUTALIAS .ram ".ram [Real Audio Media]"
TYPEOUTPUTALIAS .asf ".asf [Microsoft Advanced Streaming Format]"
TYPEOUTPUTALIAS .asx ".asx [Microsoft Advanced Streaming Extensions]"
TYPEOUTPUTALIAS .pdf ".pdf [Adobe Portable Document Format]"
TYPEOUTPUTALIAS .swf ".swf [Adobe Flash Object]"

# Microsoft Office & Pocket Office
TYPEOUTPUTALIAS .mdb ".mdb [Microsoft Access Database]"
TYPEOUTPUTALIAS .accdb ".accdb [Microsoft Access 2007 Database]"
TYPEOUTPUTALIAS .ppv ".ppv [PowerPoint Viewer File]"
TYPEOUTPUTALIAS .ppt ".ppt [PowerPoint File]"
TYPEOUTPUTALIAS .pptx ".ppxt [PowerPoint File]"
TYPEOUTPUTALIAS .xls ".xls [Excel SpreadSheet]"
TYPEOUTPUTALIAS .xlsx ".xlsx [Excel 2007 SpreadSheet]"
TYPEOUTPUTALIAS .doc ".doc [Microsoft Word Document]"
TYPEOUTPUTALIAS .docx ".docx [Microsoft Word 2007 Document]"
TYPEOUTPUTALIAS .pwd ".pwd [Microsoft Pocket Word Document]"
TYPEOUTPUTALIAS .pwt ".pwt [Microsoft Pocket Word Template]"
TYPEOUTPUTALIAS .pxl ".pxl [Microsoft Pocket Excel Document]"
TYPEOUTPUTALIAS .pxt ".pxt [Microsoft Pocket Excel Template]"
TYPEOUTPUTALIAS .rtf ".rtf [Rich Text Format]"
TYPEOUTPUTALIAS .pub ".pub [Publisher File]"
TYPEOUTPUTALIAS .mps ".mps [Microsoft Pocket Streets Map]"
TYPEOUTPUTALIAS .psm ".psm [Microsoft Pocket Automap Streets]"
TYPEOUTPUTALIAS .ics ".ics [vCalendar / iCalendar File]"

# Databases
TYPEOUTPUTALIAS .db ".db [DataBase File]"
TYPEOUTPUTALIAS .csv ".csv [CSV File]"
TYPEOUTPUTALIAS .dbf ".dbf [Database File]"

# Executables, Installations, Execution & Applications
TYPEOUTPUTALIAS .msi ".msi [Microsoft Installer Package]"
TYPEOUTPUTALIAS .cab ".cab [Cabnet Archive]"
TYPEOUTPUTALIAS .bat ".bat [Batch Files]"
TYPEOUTPUTALIAS .com ".com [Compiled Executable]"
TYPEOUTPUTALIAS .exe ".exe [Executables]"
TYPEOUTPUTALIAS .zip ".zip [Zip archives]"
TYPEOUTPUTALIAS .zip ".rar [Rar archives]"
TYPEOUTPUTALIAS .hlp ".hlp [Windows Help Files]"
TYPEOUTPUTALIAS .chm ".chm [Compiled HTML Help]"
TYPEOUTPUTALIAS .dat ".dat [Internet Explorer Installer Data File]"
TYPEOUTPUTALIAS .dll ".dll [Dynamic Link Library]"

# Misc
TYPEOUTPUTALIAS .bin ".bin [Binary File]"
TYPEOUTPUTALIAS .iso ".iso [CD-ROM/DVD-ROM Image file]"
TYPEOUTPUTALIAS .nrg ".nrg [Nero Burning ROM CD-ROM/DVD-ROM Image file]"
TYPEOUTPUTALIAS .ida ".ida [IIS default.ida - Code Red II Attack]"
TYPEOUTPUTALIAS .reg ".reg [Windows Registry META Data]"

SUBTYPE *.gz,*.Z

To save time, disk space and bandwidth it is recommended that you edit the style sheet location inside the baseline configuration to point to a common location

STYLESHEET images/analog.css

For example, you could create an administrative resource on a separate web to host these resources e.g.

STYLESHEET http://static.mycompany.com/analog.css

Note: This shared resource site will be equally useful if deploying Report Magic stats along with Analog (not covered in this guide).

Configure the Analog Site Settings

Now that you have created a common baseline for Analog’s global config, the next step is to configure Analog for each of your sites.

Navigating to d:\sites\www.domain.com create a file called analog.cfg

Open this configuration file and this time enter only the information that you removed from the baseline configuration, ensuring that you tailor the settings to this particular web site For example:

# Default Local Web Statistics Configuration

# Version 1.0.1
LOGFILE d:\sites\www.domain.com\logs\W3SVC1\*.log
OUTFILE d:\sites\www.domain.com\stats\index.html
HOSTNAME "Statistics For Domain.com"

#HOSTURL http://www.domain.com/

IMAGEDIR "images/"

# Referring URL
REFLINKINCLUDE *

# Refering Site

# Refering Site Alias
#REFALIAS http://domain.com/* http://www.domain.com/

#Request Report
REQFLOOR 10r
REQEXCLUDE *.jpg
REQEXCLUDE *.gif
REQEXCLUDE *.png
REQEXCLUDE *.bmp
REQEXCLUDE *.class
REQEXCLUDE *.js

# Custom Exclusions
FILEEXCLUDE /stats/*

You should automate the creation of this configuration file as part of your site creation activities. The fastest way to achieve this is to write the configuration file to the d:\sites\<web site>\analog.cfg as part of your website creation script.

In the above example, a default Analog CE HTML report will be generated in d:\sites\www.domain.com\stats\ (http://www.domain.com/stats/ to the user).

Ensure that you repeat the creation of this file for all web sites on your server.

Running Analog on a Schedule

Now that Analog has been configured, the last step is to schedule it to run. The simplest way to do this is to use a batch script and the Windows task scheduler.

With some creative scripting (exampled below) you can configure the scheduled job to crawl through the d:\sites folder looking for site-level analog.cfg files to parse through to the Analog CE executable.

The example below should be saved via Windows Notepad as d:\parser\GenerateStats.cmd

cls

@echo off

SET ALGROOT=d:\parser\Analog\

SET WEBROOT=d:\sitesFOR /f "tokens=*" %%A IN ('dir /b d:\sites') DO (echo Looking for: analog.cfg in %%A

IF EXIST "%WEBROOT%\%%A\analog.cfg" (
echo Found analog.cfg for %%A
md "%WEBROOT%\%%A\web\stats"
echo.
"%ALGROOT%analog.exe" +g%WEBROOT%\%%A\analog.cfg > %WEBROOT%\%%A\analog.log
echo.
) ELSE (
echo ANALOG CONFIG FILE NOT FOUND IN "%WEBROOT%\%%A\"
echo.
)
)

This script searches all sub-folders of d:\sites for the presence of an analog.cfg file e.g. d:\sites\www.mydomain.com\analog.cfg. If it finds one it ensures that there is an appropriate output directory (\web\stats) and then runs the Analog CE executable using the baseline configuration (this is implicit) as well as the local site-level analog.cfg configuration file (explicit) to produce the report.

The report will be output as specified in the site-level analog.cfg.

Any Analog parser errors will be written into the d:\sites\www.mydomain.com folder as analog.log, allowing for analysis by the client.

In Windows Task Scheduler, create a basic task that will run d:\parser\GenerateStats.cmd ever day, 12 hours, 6 hours (or at whatever interval you require).

Once complete, as you add new web sites (and their associated site-level analog.cfg files) or remove old ones they will automatically be picked-up and parsed through the statistics engine without any administrative intervention.

Tag: Scripts

Using Analog CE to provide automatic statistics on a multi-site IIS Server