2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 09:53:19 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-21 10:16:00 -05:00
2020-10-22 07:19:24 -05:00
2020-10-21 10:16:00 -05:00

Install GoAccess on Windows

  1. Install MSYS2
  2. Launch, update pacman -Syu, exit, relaunch
  3. Update any remaining packages with pacman -Su
  4. Download thepackages necessary to build GoAccess:
pacman -S git gcc base-devel ncurses ncurses-devel
  1. Grab the source code git clone https://github.com/allinurl/goaccess.git
  2. cd goaccess
  3. Create config file autoreconf -fi
  4. Configure your build (this will take a while):
./configure --enable-utf8
### NOTE
# If you want to build using MaxMind GeoIP, go to step 08
# If not, go to step 10
  1. Build libmaxminddb from release:
wget https://github.com/maxmind/libmaxminddb/releases/download/1.4.3/libmaxminddb-1.4.3.tar.gz
tar -xvf libmaxminddb-1.4.3.tar.gz
cd libmaxminddb-1.4.3
./configure
make
make install
  1. Configure for MaxMind GeoIP (this will take a while):
./configure --enable-utf8 --enable-geoip=mmdb
  1. make
  2. This will create two files: bin2c.exe and goaccess.exe
  3. In order to use these two files, you will also need 4 dll's from MSYS:
msys-2.0.dll
msys-iconv-2.dll
msys-intl-8.dll
msys-ncursesw6.dll
  1. For each release, these files will be included.
  2. Find out what kind of format your log file is: visit GoAccess for details
  3. For IIS, you can run this: scripts/iis2goaccess.sh
# Must run this in either MSYS or Git for Windows
cat u_ex201019.log | ./iis2goaccess
# Which will spit out something like:
%d %t %^ %m %U %^ %^ %^ %h %u %R %s %^ %^ %L
# OR
%d %t %^ %^ %m %U %^ %^ %^ %h %u %s %^ %^
# OR
%d %t %^ %m %r - %^ - %h %u %s %^ %T
  1. You can also retroactively pass this into your GoAccess call with:
cat  u_ex* | goaccess --log-format "$(cat u_ex170114.log | ~/goiisformat.sh)" --date-format '%Y-%m-%d' --time-format '%H:%M:%S'

Configurations

You can have configuration files so you do not have long queries in your terminal.

Simple Config file

date-format %Y-%m-%d
time-format %H:%M:%S
log-format %d %t %^ %m %U %^ %^ %^ %h %u %R %s %^ %^ %L

Better Config file

date-format %Y-%m-%d
time-format %H:%M:%S
log-format %d %t %^ %m %U %^ %^ %^ %h %u %R %s %^ %^ %L
##geoip-database GeoLite2-City.mmdb
color-scheme 3
config-dialog false
hl-header true
html-prefs {"theme":"bright","perPage":20,"layout":"horizontal","showTables":true,"visitors":{"plot":{"chartType":"bar"}}}
html-report-title GoAccess Stats
json-pretty-print true
real-os true
agent-list true
output index.html

Windows Task Scheduler

GoAccess has a built in way to auto generate reports through a websocket pointing to your webserver. For windows, this is very difficult. Instead, just create a batch file and schedule it to run every 15 minutes.

Example bat files

Example 1 - Simple

Will find today's date and grab that IIS log file to display.

@echo off
setlocal
set logdir=C:\inetpub\logs\LogFiles\W3SVC3
set wwwdir=C:\inetpub\wwwroot
for /f "tokens=1-4 delims=/-. " %%i in ('date /t') do (call :set_date %%i %%j %%k %%l)
goto :parse
:set_date
if "%1:~0,1%" gtr "9" shift
for /f "skip=1 tokens=2-4 delims=(-)" %%m in ('echo,^|date') do (set %%m=%1&set %%n=%2&set %%o=%3)
goto :eof
:parse
set fsta=u_ex
set fmid=%yy:~0,2%
set fend=%mm%%dd%.log
set fname=%fsta%%fmid%%fend%
goto :goa
:goa
goaccess -f %logdir%\%fname% -p C:\Tools\GoAccess\goa.conf >> nul 2>&1
endlocal

Example 2 - All

Will find all IIS log files from this year (will take an incredibly long time)

@echo off
setlocal
set logdir=C:\inetpub\logs\LogFiles\W3SVC3
set wwwdir=C:\inetpub\wwwroot
for /f "tokens=1-4 delims=/-. " %%i in ('date /t') do (call :set_date %%i %%j %%k %%l)
goto :parse
:set_date
if "%1:~0,1%" gtr "9" shift
for /f "skip=1 tokens=2-4 delims=(-)" %%m in ('echo,^|date') do (set %%m=%1&set %%n=%2&set %%o=%3)
goto :eof
:parse
set fsta=u_ex
set fmid=%yy:~0,2%
set fname3=%fsta%%fmid%*.log
goto :goa
:goa
type %logdir%\%fname3% | goaccess -p C:\Tools\GoAccess\goa.conf >> nul 2>&1
endlocal

Tests

I used this on a corporate server. Total requests were 5,803,378 over a period of March 01, 2020 - October 20, 2020. It aggregated 235 IIS 8.5 log files totalling ~1.11GB in file size in approximately 12 minutes and 31 seconds. An average of 7,728 lines/second.

S
Description
GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser (but for Windows)
https://github.com/allinurl/goaccess
Readme
1.4 MiB
2020-10-21 16:14:08 -05:00
Languages
C 68.8%
Roff 9.6%
Gettext Catalog 8.9%
M4 3.9%
Makefile 3.5%
Other 5.2%