Lessons Learned from Log Analysis
•Data sources are complicated
•Extensive scripting and parsing are required
•Documentation of analysis is necessary for developing methodology
•
Now that we have completed two rounds of log analysis, we have a better idea of how we can use such analysis in future to further our goal of improving the user interface for Find Articles/Find Databases/Find e-Journals.   Here are some of the lessons we have learned:

Data sources are complicated.  We created data sets from the ENCompass logs and the Apache logs.  Within the Apache logs, we had to parse data from several different files, including one for the LinkFinder Plus or Find it at Cornell data.  There are still more data sources we could pull in, including the LFP table in ENCompass, and error log files in Apache.

Without the custom scripts we had written, the kind of analysis we did would not have been possible.  Given our experience, we would like to see Endeavor consider how to make the ENCompass log data more compatible with the Apache web log data.  Additionally, if the issue with the browser back button leading to false session IDs were corrected, we could perform more meaningful and accurate analysis on some of the data in the ENCompass logs.  We would have a clearer idea of how many actual sessions were being logged and could better analyze how users move about within each session.

When we followed up with our analysis in the fall, it became clear that the only way the work would be useful to the Library would be through clear documentation of what was in fact a complicated, rather piecemeal process.  We explored using SPSS as a tool for formulating a methodology and have plans to better document the work we have done thus far.  We want to turn this data analysis product we have developed into a useful process for the Library.