This program produces web site statistics from log file records.

       Various reports are produced as a function of criteria supplied by the user.

       Depending on the time period specified by the user, one or more log files

       must be accessed.  There is always a current "access" log file and possibly

       one or more archive log files.  Archive files are gzipped and named to

       reflect the date.

       To speed processing, as little time as possible is spent reading log records.

       Fields are read into work areas and later distributed and processed as needed.

       Arrays are built driven by the needs of the various reports.  If an array fills

       up, it is enlarged using "realloc."  Arrays are sorted using quicksort after

       they are finished being populated.

       Report 1: Top n clients accessing the site.  Shows a count and the domain name.

              Sorted by count, highest first.

       Report 2: Top n files accessed.  Shows count and file (e.g. a gif or an HTML page).

              Sorted by count, highest first.

       Report 3: Files containing the string "whatever."  Shows count and client (IP).

              Sorted by count, highest first.

       Report 4: Period totals for site.  Shows individual clients, all clients, hits,

              page hits, KB transmitted.  Individual clients = unique IP's; all clients

              counts an IP as many times as days it occurs, and = the sum of the "clients"

              column in report 5; hits is sum of "hits" column in report 5; page hits =

              sum of "page hits" column in report 5; KB transmitted = sum of "KB" column

              in report 5.

       Report 5: Daily Totals for site.  Shows date, clients, hits, page hits, KB transmitted.

              Date is in form: weekday (3 char), e.g. Mon; month (3 char), day (no leading zero),

              year (4 digits).  KB is to two decimal places.

       Report 6: Daily Averages.  Shows same columns as report 5.  Values = report 4 values

              divided by number of days in period.  Clients here is based on "all clients"

              from report 4, rather than "individual clients."

       Report 7: Hourly averages.  Shows hour, hits, percentage, and KB transmitted.

              Hour appears as e.g. "Midnight to 1 am" or "1 am to 2 am,"  Percentage is

              nn.n %.

       Report 8: Summary of HTTP errors.  Shows count, error code (e.g. 404), text (e.g. Not Found).

              Sorted by count, highest first.

       Report 9: (Top 20) requests causing errors.  Shows count and request (file).

              Sorted by count, highest first.

       Design notes:

              Since this isn't Perl, and none of us can figure out how to redirect standard output

              from a shell command back to this C program's standard in, the output of the cat or

              gzcat is to a work file.

              For reports 4-9 we read in the log records one-by-one.  But we only read each

              work file once, extracting all of the data for all of those reports in one pass.

              The processing for reports 1-3 is affected by the fact that there can be from a handful

              to millions of log records to be processed for a given user submission of the stats

              engine.  This precludes building an array in-core to hold information for each ip

              encountered, e.g., without first having determined how many occur.  And even if we

              built an array based on that count of unique ip's, if we read each log record we have

              to search the array each time to find the right entry to increment the count for.

              So, a hash table is a better data structure to use.

              For these reports, we extract selected log fields from each relevant log file

              (based on the user-supplied date range) using shell commands (via system calls)

              and concatenate this output to a work file.  This allows us to determine how many

              hash table entries to allocate for the primary array.  We also can perform a number

              of useful operations on the combined log data.

              For the purposes of report 1 we "cut" extract just the first field, which is the ip,

              from each log record.*  After cutting from all relevant files, this gives us a file of

              all occurrences of ip's.  We sort this and pipe to "uniq" which gives us just the unique

              ip's that occurred.  A "wc" of that tells us how many ip's to take into account when

              allocating the primary hash table stucture.

              We can now populate the hash table.  Reading in a record from the file with all ip

              occurrences, we directly hash to the right entry and increment its count.  In case of

              the entry already being populated by a different ip (a synonym situation), we chain a

              linked list off the main entry so that each different ip mapping to that same primary

              entry gets its own entry.

              When we're done accumulating occurrences, we can produce report 1.

              For report 2 we employ a similar strategy to that used for report 1.  We extract the

              "file" field instead of the ip field.  A hash table is eventually built based on the

              file combining the "file" fields from all relevant log files.

              For report 3 we read the log records as in reports 4-9, but proceed more like we do for

              reports 1 and 2.  We look for records whose "file" field contains the user-supplied string.

              For those that do, we write a record to a file with just the ip.  After all log files

              have been read, we process the resulting ip file.  We sort it and uniq it to get the

              number of unique ip's.  We read this in and allocate an array with that many entries.

              We read the sorted ip file and accumulate the count of occurrences for each ip there.

              We then write the ip's and counts to a file and sort it by count, descending.  We read

              that in and produce the report.

              *Since for report 5 we need ip information within date, we build a work file with both

              ip and date.  We use this later for report 1, cutting the ip column to create the ip file.

*/

#include "stdio.h"

#include "string.h"

#include "time.h"

#include "math.h"

#include "stdlib.h"

#include "unistd.h"       /* For getpid, execle commands */

#include <sys/types.h>

#include <sys/socket.h>

#include <netinet/in.h>

#include <arpa/inet.h>

#include <netdb.h>

#include <ctype.h>

/* Multiplied times number of unique ip's when malloc'ing hash table */

#define scaling_factor 3

char ip[8192];            /* Host   (IP address)       */

char ident[256];       /* Ident field          */

char authuser[256];      /* Authuser             */

char timestamp1[256];    /* Time Stamp part 1 */

char timestamp2[256];    /* Time Stamp part 2 */

char file1[256];       /* HTTP Request       part 1       */

char file2[256];       /* HTTP Request       part 2       */

char file3[256];       /* HTTP Request       part 3       */

char status[256];       /* Status Code          */

long bytes;          /* Transfer Volume */

char *blank_date = "           ";     /* 11 blanks */

char  blank_ip[16];

char  blank_string[256];

char *buff;

char  buffer[1024];

char *buff2;

long  bump1 = 1000;

long  bump2 = 100;

long  bump3 = 10;

char  cat_string[256];

char  cgipath[10] = "/tmp/";

/*char  cgipath[44] = "/Space/Domains/stats.simplenet.net/cgi-bin/";*/

char  char_oldest_day[4];

char  char_oldest_year[5];

char  char_youngest_day[4];

char  char_youngest_year[5];

int   code_red_found = 0;

char  curr_file[100];

long  curr_s1 = 0;

long  curr_s2 = 0;

long  curr_s3 = 0;

char *dashed_line = "\n\t\t\t---------------------------\n\n";

char  date[12];

long  date_bias;

char  date_save[12];

char  domain[256];

char  end_date[12];

char  end_day[3];

char  end_month[3];

char  end_year[5];

long  endloop;

char  enterprise_string[64];

char *enterprise_string1 = "format=%Ses->client.ip%";

char *enterprise_string2 = "%Req->srvhdrs.content-length%";

long  entire_period = 0;

long  e400_count = 0;

char *e400_lit = "Syntax error         ";

long  e401_count = 0;

char *e401_lit = "Unauthorized         ";

long  e402_count = 0;

char *e402_lit = "Unauthorized         ";

long  e403_count = 0;

char *e403_lit = "Forbidden            ";

long  e404_count = 0;

char *e404_lit = "Not found            ";

long  e405_count = 0;

char *e405_lit = "Not found            ";

long  e406_count = 0;

char *e406_lit = "Internal server error";

long  e410_count = 0;

char *e410_lit = "No longer available  ";

long  e500_count = 0;

char *e500_lit = "Internal server error";

long  e501_count = 0;

char *e501_lit = "Not implemented      ";

long  e502_count = 0;

char *e502_lit = "Bad gateway          ";

long  e503_count = 0;

char *e503_lit = "Service unavailable  ";

long  e504_count = 0;

char *e504_lit = "Bad gateway          ";

char  error_code_file_string[256] = "error_code_file_";

char  file2_buffer[8192];

int   file10_open = 0;

int   file11_open = 0;

int   file12_open = 0;

int   file15_open = 0;

int   file16_open = 0;

long  file_date_status = 0;

char  file_list_string[256] = "file_list_";

long  fqdn_limit;

char  header_work[256];

char  header_work2[256];

long  indexx;

char *input_parm_file;

long  int_start_year;

long  int_start_month;

long  int_start_day;

long  int_end_year;

long  int_end_month;

long  int_end_day;

long  ip_date_file_removed = 0;

char  ip_date_file_string[256] = "ip_date_file_";

char  ip_save[256];

int   null_found = 0;

long  num_clients;

long  num_dates;

long  num_dates_used = 0;

long  num_uniq_items;

long  oldest_year;

long  oldest_month;

long  oldest_day;

char  outfile_string[256] = "outfile_";

char  parm_line[256];

char  pid[256] = "\0";

char  plural[2];

char  random_suffix_char[2] = " ";

int   rc = 0;

char  rec_date[12];

char  rec_day[3];

int   rec_day_int;

long  rec_hour;

char  rec_month[4];

int   rec_month_int;

long  rec_too_new;

char  rec_year[3];

int   rec_year_int;

char  rec_year_long[5];

long  record_count = 0;

char  report1_ip_file_string[256] = "report1_ip_file_";

char  report2_file_file_string[256] = "report2_file_file_";

char  report3_ip_file_string[256] = "report3_ip_file_";

char  report9_file_file_string[256] = "report9_file_file_";

char  report_file_string[256] = "report_file_";

long  report_index;

char  report_input_file[256];

long  report_limit;

char  report_lit[256];

long  report_num;

char  report_work[256];

char  r1_exclude_string[16];

long  r1_limit = 30;

char *r1_lit = "Top %ld Clients Accessing %s:\n\n";

char  r2_exclude_string[256];

long  r2_limit = 40;                  /* User supplied */

char *r2_lit = "Top %ld Files Accessed on %s:\n\n";

long  r3_limit = 40;

char *r3_lit = "Clients accessing files containing the string \"%s\" in this period:\n\n";

char  r3_string[256];

int   r5sorted_ip_date_file_created = 0;

int   r5sorted_ip_date_file_removed = 0;

char  r5sorted_ip_date_file_string[256] = "r5sorted_ip_date_file_";

long  r5_data_needed = 0;

long  r9_limit = 20;

char *r9_lit = "Top %ld Requests Causing Errors:\n\n";

char  sorted_error_code_file_string[256] = "sorted_error_code_file_";

char  sorted_report_file_string[256] = "sorted_report_file_";

char  sort_string[256];

char  start_date[12];

char  start_day[3];

char  start_month[3];

char  start_year[5];

char  std_error_file_string[256] = "std_error_file_";

char  std_error_file_string2[256] = "std_error_file2_";

long  tot_clients;

long  tot_hits;

float tot_kb;

long  tot_page_hits;

char  uniq_ip_count_string[256] = "uniq_ip_count_";

char  uniq_ip_list_string[256] = "uniq_ip_list_";

char  uniq_item_count_string[256] = "uniq_item_count_";

char  uniq_item_list_string[256] = "uniq_item_list_";

char *val;

char  work_bytes[256];

long  work_count;

char  work_date[12];

char  work_day[4];

char  work_daynum[3];

char  work_file_string[256] = "work_file_";

char  work_mon[4];

char  work_string1[256];

char  work_string2[256];

char  work_string3[256];

char  work_string4[256];

char  work_string5[256];

char  work_year[5];

long  y;

long  youngest_year;

long  youngest_month;

long  youngest_day;

typedef struct struc_1

{

       char   s1_string[256];

       long   s1_count;

       struct struc_1 *s1_next_hash_entry;

}d1;

struct struc_1 *report_hash;

struct struc_1 *report_array;      /* Really an array, not used as a hash table.  See create_report1_file(). */

typedef struct struc_3

{

       char   s3_date[12];

       long   s3_clients;

       long   s3_hits;

       long   s3_page_hits;

       float  s3_kbytes;

       struct struc_3 *s3_next_hash_entry;

}d3;

struct struc_3 *report5_hash;

typedef struct struc_4

{

       char  s4_time[21];

       long  s4_hits;

       float s4_kbytes;

}d4;

struct struc_4 *report7_array;

struct tm *time_ptr;

time_t lt;

FILE *fptr1;       /* log file list            */

FILE *fptr2;       /* cat'd log file          */

FILE *fptr3;       /* stats report file          */

FILE *fptr4;       /* count of unique ip's          */

FILE *fptr5;       /* ip from each log rec    */

FILE *fptr6;       /* report initial data as i/p */

FILE *fptr7;       /* count of report ip's or files*/

FILE *fptr8;       /* report data to sort         */

FILE *fptr9;       /* sorted report data          */

FILE *fptr10;       /* ip_date_file                */

FILE *fptr11;       /* report1_ip_file            */

FILE *fptr12;       /* report2_file_file          */

FILE *fptr13;       /* error code file          */

FILE *fptr14;       /* sorted error code file       */

FILE *fptr15;       /* report3 initial data as o/p */

FILE *fptr16;       /* report9 initial data as o/p */

FILE *fptr17;       /* input parameter file          */

long  check_date_range();

void  check_file_date();

long  check_for_match();

long  check_for_wildcards();

void  close_some_files();

int   create_record_stream();

void  create_report();

void  create_report_file();

char *dayname();

void  do_reports();

char *error_lit();

void  estimate_num_dates_entire();

void  estimate_num_dates_specific();

void  get_count();

void  get_dates();

void  get_domain_name();

void  get_form_input();

void  get_ip_count();

void  get_random_suffix();

void  get_time();

char *get_value();

long  hash_date();

long  hash_file();

long  hash_ip();

void  init();

void  init_report5_hash();

void  init_report7_array();

void  init_report_array();

void  init_report_hash();

char *month_name();

long  month_num();

void  omit_header();

void  open_report_input_file();

void  open_some_files();

void  output_report5();

long  page_hit();

void  parse_input();

void  populate_report_hash();

char *prepare_filename_string(char *);

void  prepare_filename_strings();

void  prepare_filename_suffix();

void  prepare_file_list();

void  prepare_parm_input();

void  print_glossary();

void  print_header();

void  print_report_hash();

void  print_sorry();

void  print_time();

void  report();

void  report_setup();

void  reports1and4_setup();

void  report1_setup();

void  report2_setup();

void  report3_ongoing();

void  report3_setup();

void  report4();

void  report5();

void  report5_init();

void  report5_ongoing();

void  report6();

void  report7();

void  report7_ongoing();

void  report8();

void  report9_setup();

void  reports8and9_ongoing();

void  set_num_dates();

void  update_report5_hash();

void  update_report5_totals();

void  update_report_hash();

void  update_strucs();

void  wrapup();

void  wrapup_hogs();

int main(int argc, char *argv[])

{

       input_parm_file = argv[1];

       init();

       /* get name of next log file */

       while (fscanf(fptr1,"%s",curr_file) != EOF)

       {

              if (strstr(curr_file,"%") != NULL)       {continue;}

              if (entire_period == 0)                     {check_file_date();}

              if (file_date_status != 0)           {continue;} /* too old */

              rc = create_record_stream();

              if (rc > 0) {continue;}

              omit_header();

/* to find file clobbering buffer (when "not found" becomes email address) */

printf("curr_file = %s\n",curr_file);

printf("buffer = %s\n\n",buffer);

              rec_too_new = 0;

              while (fscanf(fptr2,"%s%s%s",ip,ident,authuser)

                     != EOF && rec_too_new == 0)

              {

                     /* IP field can contain an unlimited number of leading nulls.

                        Corruption or even coring can occur.  Skip the record.

                     */

                     if (ip[0] < '!') {null_found = 1;}

                     /* authuser can sometimes consist of multiple fields,

                        so read fields til definitely at timestamp1

                     */

                     if (fscanf(fptr2,"%s",timestamp1) == EOF) {break;}

                     while (timestamp1[0] != '[') {if (fscanf(fptr2,"%s",timestamp1) == EOF) break;}

                     /* Ignore timestamp1 if it has been partially clobbered by new rec.

                        Use current ip rather than ip of new rec (hard to extract from

                        clobbered timestamp1 field).

                     */

                     if (strlen(timestamp1) != 21 || strstr(timestamp1,".") != NULL)

                     {

                            if (fscanf(fptr2,"%s%s",ident,authuser) == EOF) {break;}

                            if (fscanf(fptr2,"%s",timestamp1) == EOF) {break;}

                            while (timestamp1[0] != '[') {if (fscanf(fptr2,"%s",timestamp1) == EOF) break;}

                     }     

                     if (fscanf(fptr2,"%s",timestamp2) == EOF) {break;}

                     /* Read http fields til last char of field read is ".

                        If more than one, second is request.  There can be as

                        few as one or two fields, as many as 3 or more.

                        URL *can* contain blanks + stuff, throwing off expected field sequence.

                        For example, "GET /cgi-bin/Stats/stats.cgi [ <a href= HTTP/1.0" shows up

                        in the log record, and has 3 extraneous "fields" before "HTTP".

                      */

                     work_count = 0;

                     if (fscanf(fptr2,"%s",file2_buffer) == EOF) {break;}

                     while (file2_buffer[strlen(file2_buffer)-1] != '"')

                     {

                            work_count++;

                            if (work_count == 2)

                            {

                                   if (strlen(file2_buffer) > 255)

                                   {

                                          memcpy(file2,file2_buffer,255);

                                          memset(file2+255,'\0',1);

                                   }

                                   else {strcpy(file2,file2_buffer);}

                            }

                            if (fscanf(fptr2,"%s",file2_buffer) == EOF) {break;}

                     }

                     if (fscanf(fptr2,"%s%s",status,work_bytes) == EOF) {break;}

                     if   (strcmp(work_bytes,"-") == 0)

                            {bytes = 0;}

                     else       {bytes = atol(work_bytes);}

/*printf("%s\n",timestamp1);

*/

                     if       (strstr(file2,"default.ida") != NULL)       {code_red_found = 1;}

                     if       (null_found == 0 && code_red_found == 0)

                     {

                            update_strucs();

                     }

                     else

                     {

                            null_found = 0;

                            code_red_found = 0;

                     }

              }

              strcpy(work_string1,"rm ");

              strcat(work_string1,work_file_string);

              system(work_string1);

       }

/*       strcpy(work_string1,"rm ");

       strcat(work_string1,file_list_string);

       system(work_string1);

*/

       close_some_files();

       do_reports();

       wrapup();

/*       print_time("ending");

*/

       return (0);

}

void init()

{

/*       print_time("starting");

*/

       prepare_parm_input();

       prepare_filename_suffix();

       prepare_filename_strings();

       prepare_file_list();

       open_some_files();

       if (entire_period == 0) {get_dates();}

       set_num_dates();

       init_report5_hash();

       init_report7_array();

       memset(blank_string,' ',255);

       memset(blank_string+255,'\0',1);

       strcpy(report_work,blank_string);

       memset(blank_ip,' ',15);

       memset(blank_ip+15,'\0',1);

       strcpy(domain,blank_string);

       strcpy(domain,get_value("login"));

       strcpy(r3_string,get_value("nameaccessfile"));

}

void get_time()

{

       lt = time(NULL);

       time_ptr = localtime(&lt);

}

void print_time(char * msg)

{

       get_time();

       printf("%s at %s<br>",msg,asctime(time_ptr));

}

void prepare_parm_input()

{

       /*

              Input parameters containing stats form fields and the path to the

              stats data are contained in the input parm file provided by the

              stats engine wrapper.  Open the file, read all the records, and

              concatenate the data into one long string.  This string will be

              equivalent to a query string that has been url-decoded.

              The get_value subroutine will take this string and extract values

              for specified input variables.

       */

       if ((fptr17 = fopen(input_parm_file,"r")) == NULL  ) 

       {

              printf("can't open %s\n",input_parm_file);

              exit(1);

       }

       strcpy(buffer,"\0");

       while (fscanf(fptr17,"%s",parm_line) != EOF)

       {

              strcat(buffer,parm_line);

              strcat(buffer,"&");

       }

       if (strcmp(get_value("period"),"entire") == 0)       {entire_period = 1;}

       if (strcmp(get_value("usedailytotal"),"on") == 0               /* if report 5 requested */

              || strcmp(get_value("useperiodtotal"),"on") == 0        /* or report 4 requested */

              || strcmp(get_value("usedailyavg"),"on") == 0)         /* or report 6 requested */

       {r5_data_needed = 1;}

}

void prepare_filename_suffix()

{

       strcat(pid,get_value("login"));         /* use for unique filenames */

       strcat(pid,"_");

       get_random_suffix();

       /* Shouldn't have more than one for an account at a time, but just in case */

       strcat(pid,random_suffix_char);         /* use for unique filenames */

}

void get_random_suffix()

{

       int stime, random_suffix;

       long ltime;

       ltime = time(NULL);

       stime = (unsigned) ltime/2;

       srand(stime);

       random_suffix = rand()%10;

       switch (random_suffix)

       {

              case 0: strcpy(random_suffix_char,"0"); break;

              case 1: strcpy(random_suffix_char,"1"); break;

              case 2: strcpy(random_suffix_char,"2"); break;

              case 3: strcpy(random_suffix_char,"3"); break;

              case 4: strcpy(random_suffix_char,"4"); break;

              case 5: strcpy(random_suffix_char,"5"); break;

              case 6: strcpy(random_suffix_char,"6"); break;

              case 7: strcpy(random_suffix_char,"7"); break;

              case 8: strcpy(random_suffix_char,"8"); break;

              case 9: strcpy(random_suffix_char,"9"); break;

              default:                          break;

       }

}

void prepare_filename_strings()

{

       strcpy(error_code_file_string,prepare_filename_string(error_code_file_string));

       strcpy(file_list_string,prepare_filename_string(file_list_string));

       strcpy(ip_date_file_string,prepare_filename_string(ip_date_file_string));

       strcpy(outfile_string,prepare_filename_string(outfile_string));

       strcpy(r5sorted_ip_date_file_string,prepare_filename_string(r5sorted_ip_date_file_string));

       strcpy(report_file_string,prepare_filename_string(report_file_string));

       strcpy(report1_ip_file_string,prepare_filename_string(report1_ip_file_string));

       strcpy(report2_file_file_string,prepare_filename_string(report2_file_file_string));

       strcpy(report3_ip_file_string,prepare_filename_string(report3_ip_file_string));

       strcpy(report9_file_file_string,prepare_filename_string(report9_file_file_string));

       strcpy(sorted_error_code_file_string,prepare_filename_string(sorted_error_code_file_string));

       strcpy(sorted_report_file_string,prepare_filename_string(sorted_report_file_string));

       strcpy(uniq_ip_count_string,prepare_filename_string(uniq_ip_count_string));

       strcpy(uniq_ip_list_string,prepare_filename_string(uniq_ip_list_string));

       strcpy(uniq_item_count_string,prepare_filename_string(uniq_item_count_string));

       strcpy(uniq_item_list_string,prepare_filename_string(uniq_item_list_string));

       strcpy(work_file_string,prepare_filename_string(work_file_string));

}

char *prepare_filename_string(char *filename_string)

{

       strcpy(work_string1,cgipath);

       strcat(work_string1,filename_string);

       strcat(work_string1,pid);

       return(work_string1);

}

void prepare_file_list()

{

       strcpy(work_string1,"ls ");

/*       strcat(work_string1,get_value("path"));

*/       strcat(work_string1,"*access* > ");

       strcat(work_string1,file_list_string);

       system(work_string1);

}

void open_some_files()

{

       if ((fptr1  = fopen(file_list_string,"r")) == NULL)          {printf("can't open file_list\n");exit(1);}

       if ((fptr3  = fopen(outfile_string,"w")) == NULL)             {printf("can't open outfile\n");exit(1);}

       if ((fptr15 = fopen(report3_ip_file_string,"w")) == NULL)       {printf("can't open report3_ip_file\n");exit(1);}

       if ((fptr16 = fopen(report9_file_file_string,"w")) == NULL)       {printf("can't open report9_file_file\n");exit(1);}

       if (r5_data_needed == 1)

       {

              if ((fptr10 = fopen(ip_date_file_string,"w")) == NULL)

              {

                     printf("can't open ip_date_file\n");

                      exit(1);

              }

              file10_open = 1;

       }

       else if (strcmp(get_value("useclients"),"on") == 0)         /* if report 1 requested */

       {

              if ((fptr11 = fopen(report1_ip_file_string,"w")) == NULL)

              {

                     printf("can't open report1_ip_file\n");

                      exit(1);

              }

              file11_open = 1;

       }

       if (strcmp(get_value("usefiles"),"on") == 0)            /* if report 2 requested */

       {

              if ((fptr12 = fopen(report2_file_file_string,"w")) == NULL)

              {

                     printf("can't open report2_file_file\n");

                      exit(1);

              }

              file12_open = 1;

       }

}

void set_num_dates()

{

       if       (entire_period == 0)       {estimate_num_dates_specific();}

       else                        {estimate_num_dates_entire();}

}

void estimate_num_dates_specific()

{

       /*

              6/98 - 6/98 ->  1 month

              6/97 - 6/98 -> 13 months

              7/97 - 6/98 -> 12 months

              5/97 - 6/98 -> 14 months

       */

       long work;

       int year_diff, month_diff, startmonth, endmonth;

       /* find number of months in range, multiply by 31 */

       year_diff = atoi(end_year) - atoi(start_year);

       startmonth = atoi(start_month);

       endmonth   = atoi(end_month);

       month_diff = endmonth - startmonth;

       num_dates  = 31 * ((12 * year_diff) + month_diff + 1);

       /* find date_bias (see comment in hash_date subroutine) */

       memcpy(rec_year_long,start_year,4);

       memset(rec_year_long + 4,'\0',1);

       strcpy(rec_month,start_month);

       strcpy(rec_day,"01");

       work = (long) (372 * atoi(rec_year_long) + 31 * (atoi(rec_month) - 1) + atoi(rec_day));

       date_bias = work % (long) num_dates;

}

long month_num(char *month_nam)

{

       if (strcmp("Jan",month_nam) == 0) {return (1);}

       if (strcmp("Feb",month_nam) == 0) {return (2);}

       if (strcmp("Mar",month_nam) == 0) {return (3);}

       if (strcmp("Apr",month_nam) == 0) {return (4);}

       if (strcmp("May",month_nam) == 0) {return (5);}

       if (strcmp("Jun",month_nam) == 0) {return (6);}

       if (strcmp("Jul",month_nam) == 0) {return (7);}

       if (strcmp("Aug",month_nam) == 0) {return (8);}

       if (strcmp("Sep",month_nam) == 0) {return (9);}

       if (strcmp("Oct",month_nam) == 0) {return (10);}

       if (strcmp("Nov",month_nam) == 0) {return (11);}

       if (strcmp("Dec",month_nam) == 0) {return (12);}

/*       printf("%s %s bad month name = %s<br>",get_value("login"),timestamp1,month_name);

*/     return (13);

}

char * month_name(long month_number)

{

       if (month_number == 1)  {return "Jan";}

       if (month_number == 2)  {return "Feb";}

       if (month_number == 3)  {return "Mar";}

       if (month_number == 4)  {return "Apr";}

       if (month_number == 5)  {return "May";}

       if (month_number == 6)  {return "Jun";}

       if (month_number == 7)  {return "Jul";}

       if (month_number == 8)  {return "Aug";}

       if (month_number == 9)  {return "Sep";}

       if (month_number == 10) {return "Oct";}

       if (month_number == 11) {return "Nov";}

       if (month_number == 12) {return "Dec";}

/*       printf("bad month number = %s<br>",month_num);

*/     return "Jan";

}

void estimate_num_dates_entire()

{

       /*

       For "entire" period, allow for two full years of dates.  Since the most

       recent date is yesterday, the date range begins two years ago today.

       To find the date_bias (see hash_date subroutine for comments), just find the

       number of days so far this year.

       Why is it so simple?  Suppose today is 9/16/98.  The normal ("specific")

       date bias calculation would produce "work" = (365 * 98) + (31 * 9) + 16 = 36065,

       which would be reduced to 295 via modulo 730 (730 is num_dates, 2 * 365).

       295 just happens to be 31*9 + 16, the number of days so far this year.

       So we don't need the year, nor do we need to modulo.

       p.s. Don't sweat leap year.

       */

       num_dates = 744; /* Since 372 used in hash_date, double it here, else bad things. */

/*       num_dates = 730;

*/

       get_time();

       date_bias = (long) (31 * (time_ptr->tm_mon) + time_ptr->tm_mday);

/*       date_bias = (long) (31 * (time_ptr->tm_mon+1) + time_ptr->tm_mday);

*/

}

void get_dates()

{

       strcpy(start_day,  get_value("startday"));

       strcpy(start_month,get_value("startmonth"));

       strcpy(start_year, get_value("startyear"));

       strcpy(end_day,    get_value("endday"));

       strcpy(end_month,  get_value("endmonth"));

       strcpy(end_year,   get_value("endyear"));

       int_start_year       = atoi(start_year);

       int_start_month       = atoi(start_month);

       int_start_day       = atoi(start_day);

       int_end_year = atoi(end_year);

       int_end_month       = atoi(end_month);

       int_end_day  = atoi(end_day);

       strcpy(start_date,month_name(int_start_month));

       strcat(start_date," ");

       strcat(start_date,start_day);

       strcat(start_date," ");

       strcat(start_date,start_year);

       strcpy(end_date,month_name(int_end_month));

       strcat(end_date," ");

       strcat(end_date,end_day);

       strcat(end_date," ");

       strcat(end_date,end_year);

}

void init_report5_hash()

{

       register long i;

       if ((report5_hash = (struct struc_3 *) malloc(num_dates * sizeof(struct struc_3))) == NULL)

       {

              printf("error allocating report5 hash - aborting");

              exit(1);

       }

       for (i = 0; i < num_dates; i++)

       {

              strcpy(report5_hash[i].s3_date,blank_date);

              report5_hash[i].s3_clients              = 0;

              report5_hash[i].s3_hits                 = 0;

              report5_hash[i].s3_page_hits            = 0;

              report5_hash[i].s3_kbytes        = 0;

              report5_hash[i].s3_next_hash_entry     = NULL;

       }

}

void init_report7_array()

{

       long i;

       if ((report7_array = (struct struc_4 *) malloc(24 * sizeof(struct struc_4))) == NULL)

       {

              printf("error allocating report7_array - aborting");

              exit(1);

       }

       for (i = 0; i < 24; i++)

       {

              report7_array[i].s4_hits = 0;

              report7_array[i].s4_kbytes = 0;

       }

       strcpy(report7_array[0].s4_time, "Midnight to   1 AM  ");

       strcpy(report7_array[1].s4_time, "  1 AM   to   2 AM  ");

       strcpy(report7_array[2].s4_time, "  2 AM   to   3 AM  ");

       strcpy(report7_array[3].s4_time, "  3 AM   to   4 AM  ");

       strcpy(report7_array[4].s4_time, "  4 AM   to   5 AM  ");

       strcpy(report7_array[5].s4_time, "  5 AM   to   6 AM  ");

       strcpy(report7_array[6].s4_time, "  6 AM   to   7 AM  ");

       strcpy(report7_array[7].s4_time, "  7 AM   to   8 AM  ");

       strcpy(report7_array[8].s4_time, "  8 AM   to   9 AM  ");

       strcpy(report7_array[9].s4_time, "  9 AM   to  10 AM  ");

       strcpy(report7_array[10].s4_time," 10 AM   to  11 AM  ");

       strcpy(report7_array[11].s4_time," 11 AM   to  12 PM  ");

       strcpy(report7_array[12].s4_time," 12 PM   to   1 PM  ");

       strcpy(report7_array[13].s4_time,"  1 PM   to   2 PM  ");

       strcpy(report7_array[14].s4_time,"  2 PM   to   3 PM  ");

       strcpy(report7_array[15].s4_time,"  3 PM   to   4 PM  ");

       strcpy(report7_array[16].s4_time,"  4 PM   to   5 PM  ");

       strcpy(report7_array[17].s4_time,"  5 PM   to   6 PM  ");

       strcpy(report7_array[18].s4_time,"  6 PM   to   7 PM  ");

       strcpy(report7_array[19].s4_time,"  7 PM   to   8 PM  ");

       strcpy(report7_array[20].s4_time,"  8 PM   to   9 PM  ");

       strcpy(report7_array[21].s4_time,"  9 PM   to  10 PM  ");

       strcpy(report7_array[22].s4_time," 10 PM   to  11 PM  ");

       strcpy(report7_array[23].s4_time," 11 PM   to Midnight");

}

void check_file_date()

{

       char *date_pointer;

       char  file_year[3];

       int   file_year_int;

       char  file_month[3];

       int   file_month_int;

       char  file_day[3];

       int   file_day_int;

       if ((date_pointer = strstr(curr_file,"-")) == NULL)  /* can't tell */

       {file_date_status = 0; return;}

       memcpy(file_year,date_pointer+1,2);

       memset(file_year+2,'\0',1);

       file_year_int = atoi(file_year);

       if        (file_year_int > 90)       {file_year_int += 1900;}

       else                        {file_year_int += 2000;}

       memcpy(file_month,date_pointer+3,2);

       memset(file_month+2,'\0',1);

       file_month_int = atoi(file_month);

       memcpy(file_day,date_pointer+5,2);

       memset(file_day+2,'\0',1);

       file_day_int = atoi(file_day);

       if (file_year_int < atoi(start_year)) {file_date_status = -1; return;}

       if (file_year_int == atoi(start_year))

       {

              if ((file_month_int < atoi(start_month))

              ||  (file_month_int == atoi(start_month) && file_day_int < atoi(start_day)))

              { file_date_status = -1; return;}

       }

       file_date_status = 0;

}

int create_record_stream()

{

       strcpy(cat_string,blank_string);

       if        (strstr(curr_file,".gz") != NULL)

       {

              strcpy(cat_string,"/usr/bin/gzcat ");

       }

       else

       {

              strcpy(cat_string,"cat ");

       }

       strcat(cat_string,curr_file);

       strcat(cat_string," >");

       strcat(cat_string,work_file_string);   /* overwrite work file if it exists */

       system(cat_string);

       if ((fptr2 = fopen(work_file_string,"r")) == NULL)

       {

              printf("can't open work_file\n");

               exit(1);

       }

       return 0;

}

void omit_header()

{

       /* Enterprise server puts a one-line header at the start of a log file.

          If it's there, omit it.

       */

       fscanf(fptr2,"%s",header_work);

       if(strcmp(header_work,"") == 0) /* Empty file */

       {return;}

       if (strstr(header_work,"format=%Ses->client.ip%") != NULL)

       {

              fscanf(fptr2,"%s%s%s%s%s%s",header_work,header_work,header_work,header_work,header_work,header_work);

       }

       else

       {

              fscanf(fptr2,"%s",header_work2);       /* Don't know why it would, but...*/

              if (strcmp(header_work2,"-") != 0)  /* If *2nd* fscanf got ip field */

              {

                     strcpy(ip,header_work2);

                     fscanf(fptr2,"%s",ident);

              }

              else

              {

                     strcpy(ip,header_work);

                     strcpy(ident,header_work2);

              }

              fscanf(fptr2,"%s",authuser);

              /* authuser can sometimes consist of multiple fields,

                 so read fields til definitely at timestamp1

               */

              fscanf(fptr2,"%s",timestamp1);

              while (timestamp1[0] != '[')

              { fscanf(fptr2,"%s",timestamp1);}

              fscanf(fptr2,"%s",timestamp2);

              /* Read http fields til last char of field read is ".

                 If more than one, second is request.  There can be as

                 few as one or two fields, as many as 3 or more.

                 URL *can* contain blanks + stuff, throwing off expected field sequence.

                 For example, "GET /cgi-bin/Stats/stats.cgi [ <a href= HTTP/1.0" shows up

                 in the log record, and has 3 extraneous "fields" before "HTTP".

               */

              work_count = 0;

              fscanf(fptr2,"%s",file2_buffer);

              while (file2_buffer[strlen(file2_buffer)-1] != '"')

              {

                     work_count++;

                     if (work_count == 2)

                     {

                            if (strlen(file2_buffer) > 255)

                            {

                                   memcpy(file2,file2_buffer,255);

                                   memset(file2+255,'\0',1);

                            }

                            else {strcpy(file2,file2_buffer);}

                     }

                     fscanf(fptr2,"%s",file2_buffer);

              }

              fscanf(fptr2,"%s%s",status,work_bytes);

              if   (strcmp(work_bytes,"-") == 0)

                     {bytes = 0;}

              else       {bytes = atol(work_bytes);}

              if       (strstr(file2,"default.ida") != NULL)       {code_red_found = 1;}

              if       (null_found == 0 && code_red_found == 0)

              {

                     update_strucs();

              }

              else

              {

                     null_found = 0;

                     code_red_found = 0;

              }

       }

}

void update_strucs()

{

       long i;

       parse_input();

       if        (entire_period == 0)     {i = check_date_range();}

       else                        {i = 0;}

       if        (i < 0)       {return;}                  /* rec too old */

       else if     (i > 0)       {rec_too_new = 1; return;}       /* rec too new, done with file */

       if (atoi(status) > 399)       {reports8and9_ongoing(); return;}

       record_count++;

/*     if (record_count > 1000000)

       {

              wrapup_hogs();*/ /* shunt request to hogs queue */

/*            printf("hog found -- %s\n",get_value("login"));

              exit(10);

       }

*/

       /* cut recs for report input files, if reports requested */

       if (r5_data_needed == 1)

       {

              fprintf(fptr10,"%s %s\n",ip,rec_date);

              report5_ongoing();         /* needed if r4, r5, or r6 requested */

       }

       else if (strcmp(get_value("useclients"),"on") == 0)  /* if report 1 requested */

       {

              fprintf(fptr11,"%s\n",ip);

       }

       if (strcmp(get_value("usefiles"),"on") == 0)            /* if report 2 requested */

       {

              fprintf(fptr12,"%s\n",file2);

       }

       if (strcmp(get_value("usehourly"),"on") == 0)            {report7_ongoing();}

       if (strcmp(get_value("useclientaccess"),"on") == 0)     {report3_ongoing();}

}

long check_date_range()

{

       /* If log rec date before start date */

       if ((rec_year_int < int_start_year)

       || (rec_year_int == int_start_year && rec_month_int < int_start_month)

       || (rec_year_int == int_start_year && rec_month_int == int_start_month && rec_day_int < int_start_day))

       {

              return -1;

       }

       /* If log rec date after end date */

       else if ((rec_year_int > int_end_year)

       || (rec_year_int == int_end_year && rec_month_int > int_end_month)

       || (rec_year_int == int_end_year && rec_month_int == int_end_month && rec_day_int > int_end_day))

       {

              return 1;

       }

       else return 0;

}

void parse_input()

{

       char work_hour[3];

       memcpy(rec_date,timestamp1+1,11);

       memset(rec_date + 11,'\0',1);

       memcpy(work_hour,timestamp1+13,2);

       memset(work_hour + 2,'\0',1);

       rec_hour = atoi(work_hour);

       memcpy(rec_year_long,rec_date+7,4);

       memset(rec_year_long+4,'\0',1);

       rec_year_int = atoi(rec_year_long);

       memcpy(rec_month,rec_date+3,3);

       memset(rec_month+3,'\0',1);

       rec_month_int = month_num(rec_month);

       memcpy(rec_day,rec_date,2);

       memset(rec_day+2,'\0',1);

       rec_day_int = atoi(rec_day);

       if (record_count > 0)

       {

              if (rec_year_int  < oldest_year

              || (rec_year_int == oldest_year && rec_month_int  < oldest_month)

              || (rec_year_int == oldest_year && rec_month_int == oldest_month && rec_day_int < oldest_day))

              {

                     oldest_year         = rec_year_int;

                     oldest_month        = rec_month_int;

                     oldest_day          = rec_day_int;

                     strcpy(char_oldest_year,rec_year_long);

                     strcpy(char_oldest_day,rec_day);

              }

              if (rec_year_int  > youngest_year

              || (rec_year_int == youngest_year && rec_month_int  > youngest_month)

              || (rec_year_int == youngest_year && rec_month_int == youngest_month && rec_day_int > youngest_day))

              {

                     youngest_year              = rec_year_int;

                     youngest_month             = rec_month_int;

                     youngest_day        = rec_day_int;

                     strcpy(char_youngest_year,rec_year_long);

                     strcpy(char_youngest_day,rec_day);

              }

       }

       else

       {

              oldest_year         = rec_year_int;

              oldest_month        = rec_month_int;

              oldest_day          = rec_day_int;

              strcpy(char_oldest_year,rec_year_long);

              strcpy(char_oldest_day,rec_day);

              youngest_year              = rec_year_int;

              youngest_month             = rec_month_int;

              youngest_day        = rec_day_int;

              strcpy(char_youngest_year,rec_year_long);

              strcpy(char_youngest_day,rec_day);

       }

}

void close_some_files()

{

       if (file10_open) {fclose(fptr10);}

       if (file11_open) {fclose(fptr11);}

       if (file12_open && fptr12 != fptr11) {fclose(fptr12);}

       fclose(fptr15);

       if (fptr16 != fptr6) {fclose(fptr16);}

}

void do_reports()

{

       print_header();

       if (record_count == 0)       {print_sorry(); return;}

       print_time("starting report 5");

       if (r5_data_needed == 1)                           {report5();}

       if (strcmp(get_value("useperiodtotal"),"on") == 0       /* report 4 requested */

       ||  strcmp(get_value("useclients"),"on") == 0)            /* report 1 requested */

                                                        {reports1and4_setup();}

       print_time("starting report 4");

       if (strcmp(get_value("useperiodtotal"),"on") == 0)       {report4();}

       print_time("starting report 6");

       if (strcmp(get_value("usedailyavg"),"on") == 0)         {report6();}

       print_time("starting report 7");

       if (strcmp(get_value("usehourly"),"on") == 0)            {report7();}

       print_time("starting report 1");

       if (strcmp(get_value("useclients"),"on") == 0)         {report_num = 1; report();}

       print_time("starting report 2");

       if (strcmp(get_value("usefiles"),"on") == 0)            {report_num = 2; report();}

       print_time("starting report 3");

       if (strcmp(get_value("useclientaccess"),"on") == 0)       {report_num = 3; report();}

       print_time("starting report 8");

                                                        {report8();}