Logo 
Search:

Sharepoint Forum

Ask Question   UnAnswered
Home » Forum » Sharepoint       RSS Feeds

SharePoint Usage Analysis

  Asked By: Wendy    Date: Apr 03    Category: Sharepoint    Views: 1781

i was wondering.. about following post

----------
I figured out the SPS/WSS log file parsing problem I was having. The
MSDN article describing the format of th log file is MOSTLY right.

When I tried to read each log entry header field individually, the C#
program crashed and burned. I found five extra bytes within the fixed
sized header. After looking at a hex dump of the file and a disassembly
of the C# program, it hit me. The C++ compiler inserted extra bytes to
keep struct fields on word and dword boundaries. When SPS writes out
the struct, it includes the extra 5 fill bytes. SharePoint code did not
declare the struct fields to be packed, therefore the extra bytes
inserted for performance purposes.

Not having a lot of time to discover the "perfect and proper" layout
attributes, I used explicit struct layout to mimic the log entry header.
It also requires using the Marshal object to work with unmanaged memory.
----------

Whether or not a "conversion utility" existed as of yet for sts log parsing?

Helpful links

1) weblogs.asp.net/.../409614.aspx

2) msdn.microsoft.com/.../...WSSUsageEventLogging.asp

Share: 

 

24 Answers Found

 
Answer #1    Answered By: Harshita Padwal     Answered On: Apr 03

You can check the SQL Server Reporting Service Pack for Sharepoint.

www.microsoft.com/.../details.aspx

It comes with log  parser/extractor that can be automated to load your IIS logs and STS logs into a SQL database for easy reporting and analysis

 
Answer #2    Answered By: Jennifer Jones     Answered On: Apr 03

do you know if this is even supported by microsoft?

well following the documentation doesnt seem to work  at least with wss/sps sp2? IIS parsing  seems to work fine though...

C:\Program Files\Report Packs\SPS\Data Extraction>RPDataExtraction.exe
Resolving...Microsoft.SharePoint.Portal.resources, Version=11.0.0.0, Culture=en-
US, PublicKeyToken=71e9bce111e9429c
Resolving...Microsoft.SharePoint.Portal.resources, Version=11.0.0.0, Culture=en,
PublicKeyToken=71e9bce111e9429c
Parsing IIS log  files...

Parsing C:\Program Files\Report Packs\SPS\Data Extraction\IIS\HOUSTON\W3SVC1 with Log Parser...finished

Parsing WSS Log files...

Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\HOUSTON...
An exception has occured:
System.UnauthorizedAccessException
Access to the path "C:\Program Files\Report Packs\SPS\Data Extraction\WSS\HOUSTON" is denied.

-so i deviate from the documentation and just place the sts  logs directly under the WSS root (rather than creating a SERVERNAME folder) - then this happens.....

C:\Program Files\Report Packs\SPS\Data Extraction>RPDataExtraction.exe
Resolving...Microsoft.SharePoint.Portal.resources, Version=11.0.0.0, Culture=en-
US, PublicKeyToken=71e9bce111e9429c
Resolving...Microsoft.SharePoint.Portal.resources, Version=11.0.0.0, Culture=en,
PublicKeyToken=71e9bce111e9429c
Parsing IIS Log files...

Parsing C:\Program Files\Report Packs\SPS\Data Extraction\IIS\HOUSTON\W3SVC1 with Log Parser...finished

Parsing WSS Log files...

Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\00.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\01.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\02.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\03.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\04.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\07.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\09.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\10.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\12.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\13.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\14.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\15.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\16.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\19.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\22.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\23.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\24.log...finished
Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\25.log...
An exception has occured:
System.Data.SqlClient.SqlException
Error converting data type nvarchar to smalldatetime.

C:\Program Files\Report Packs\SPS\Data Extraction>

and i thought it was the answer to all my problems...

 
Answer #3    Answered By: Annie Norris     Answered On: Apr 03

It should be supported because Microsoft built it.

Not sure about SP2 though, most of our client are still using the Reporting pack with SP1. I can check and get back to you on SP2 support.

In your case it looks like the extraction stops on 25.log and seems to have an issue converting a value to a date.

 
Answer #4    Answered By: Chadd Hahn     Answered On: Apr 03

yes but doing it the way that i did, well i went against the documentation.. look at the path of the log  files.. doesnt match the documentation: it should be working on the first attempt C:\Program Files\Report Packs\SPS\Data Extraction\WSS\HOUSTON - where a directory with a GUID (in my case 6A4C3822-7B8B-473D-9C63-7AE00F38D36D) contains folders with dates and then daily log files inside those folders (in my case 30). i went the whole NTFS security problem  route and completely relaxed folder permissions on the WSS directory even and didnt change any of the symptoms.

i really loathe calling microsoft.. i checked out the GDN workspace and there is sourcecode there, too bad i dont know c# enough to know whats goin on.

Excerpt from documentation in the MSI:
Copying the IIS and Windows SharePoint Services log files
The WSS and IIS key attributes in the RPDataExtraction.exe.config file  are relative paths to the Data Extraction Program (DEP) location. By default, the DEP is installed under %Program Files%\Report Packs\SPS\Data Extraction. The value for the IIS key is the location of the sub-folder where the IIS logs are copied. Under the IIS folder, you need to create a sub-folder with machine names of each front-end web server in the farm. Underneath each machine name, create a sub-folder name for each IIS Website identifier starting with W3SVC*. The value for the WSS key is the name of the sub-folder where the Windows SharePoint Services logs are copied. Underneath the Windows SharePoint Services folder, you need to create a subfolder name with machine names of each front-end Web server. Make sure the IIS log folders from the other machines in the farm are the same name as the log folder name on the machine that the DEP is running on.

Here is an example:

SERVER1 – Front-end web server running DEP

SERVER2 – Front-end web server

SERVER3 – Front-end web server

SERVER1 has an IIS log folder called W3SVC1051025 for a virtual server

SERVER2 has an IIS log folder called W3SVC7984271 for a virtual server

SERVER3 has an IIS log folder called W3SVC1879412 for a virtual server

The folder structure under the DEP folder should be structured as follows:

/IIS
/SERVER1

/W3SVC1051025

/SERVER2

/W3SVC1051025

/SERVER3

/W3SVC1051025

/WSS

/SERVER1

/SERVER2

/SERVER3

 
Answer #5    Answered By: Laura Walker     Answered On: Apr 03

the offending line(s) in the error when i hack it to process the sts  log
files is the request to the Central Admin Website.

im dedicated to getting this working, or at least confirming that im not the
only one with the problem. i just tried the same process with a SPS/WSS sp1
Fresh Portal environment and got the same symptoms. if i have to sit there and
manually edit out lines in the stslog and deviate from documentation and place
the sts log  files at the root of the "WSS" directory(which would also mean that
i cant place more than 1 day without writing yet another script to rename the
00.log to something unique), well.. theres got to be a fix for this codewise!

 
Answer #6    Answered By: Titus Sweet     Answered On: Apr 03

the workspace the the DEP utility  is
www.gotdotnet.com/.../workspace.aspx\
d79a7d21a6

 
Answer #7    Answered By: Jagdish Joshi     Answered On: Apr 03

does anyone else confirm that this application works? im having a heck of a time  getting it working by following the documentation.

 
Answer #8    Answered By: Christop Mcfadden     Answered On: Apr 03

I've been able to get it working after a fashion... can only get a couple of the reports to work  and I am not entirely confident the figures are correct.

 
Answer #9    Answered By: Victoria Bell     Answered On: Apr 03

We installed it and configured it. some of the reports work  and others don't. We haven't been able to get any page reports but we have gotten storage reports and site/area reports. We were looking at the report files when we were required to changed all server and service passwords in our Domain. That broken the reporting service and as best as we can determine we will have to reinstall and rebuild the service and parse all of the logs again to get it to work again. Then it will probably break again in 90 days when we are required to change passwords again.

I think we stick with ANALOG run against the IIS log  and mwe are looking at the Summary_Net product ( http://www.summary.net/ )

 
Answer #10    Answered By: Cassidy Sharpe     Answered On: Apr 03

our problem  is with the report server service account serialization.. what a lovely problem isnt it? the funniest thing about report server is that there were books written about it before it ever even became RC2 quality!

it really bothers me that i have no suitable tool to combine my IIS/WSS logs though. i use analog now but the management team is asking for global reports of the SPS environment..i cant just stop at the dll requests.

so far i have deduced that the logparsing will break 2 ways by following the documentation.
-the first way is if there is a call to the "central admin" app on your portal (i.e. a call to a different port seems to mess up thier parsing  engine)
-the second and most dissapointing problem is that i must place the sts  (00.log, 01.log and so on) directly in the ROOT of the <add key="WSSPath" value="WSS"/> directory. when i use the example batch file, well i get an "access denied error" when trying to go into that folder returned by the DEP because the example batch file  copies the GUID'd directory to that WSS path (and then has of course a seperate folder for each day which also cannot be placed in the root of the WSS path).

this is the most annoyed ive been with sharepoint  in a LONG time. days like this i wish i was back in plumtree land. when management sees something i can download from microsoft, they assume FULL support for it through microsoft  is provided. now i gotta cash this check, its been 6 days since any replys and still have no assistance from the gotdotnet site.]

-must stop ranting now

 
Answer #11    Answered By: Linda Mason     Answered On: Apr 03

I have it working and have even customized it!

The SQL Data exception is a known bug: support.microsoft.com/default.aspx

The supplied batch file  also works for me. The pertinent lines from my file:
ECHO D | XCOPY /S /Y /O /K C:\WINDOWS\system32\LogFiles\W3SVC1 IIS\intranet\W3SVC1
ECHO D | XCOPY /S /Y /O /K C:\WINDOWS\system32\LogFiles\STS WSS\intranet

 
Answer #12    Answered By: Hans Weiss     Answered On: Apr 03

well i havent had a successfull DEP as of yet. what is your
OS/SQL/SPS/WSS Service Pack level?

ill try adding another folder to the

"ECHO D | XCOPY /S /Y /O /K C:\WINDOWS\system32\LogFiles\STS WSS\intranet"

as mine was copying to the already created <WSSPath> like so

"ECHO D | XCOPY /S /Y /O /K C:\WINDOWS\system32\LogFiles\STS WSS"

curios as well, whats your config read  as the <WSSPath>?

 
Answer #13    Answered By: Alison West     Answered On: Apr 03

WSSPath is "WSS"

But, underneath the WSSPath you need a folder with the server name (intranet in my example).

 
Answer #14    Answered By: Freddy Heath     Answered On: Apr 03

yeah did that too as stated in documentation and it still didnt work.. not too much assistance from microsoft  on this either.

ill try again regardless as now this is something that management simply must have!

 
Answer #15    Answered By: Joanna Dixon     Answered On: Apr 03

The exe, config and bat files should all be in the same directory. The IISPath and WSSPath folders should be below the exe/config/bat directory.

Give a shout if you have more troubles.

 
Answer #16    Answered By: Damon Garner     Answered On: Apr 03

i re-read the documentation. i took into account that everyone but me can get it working by following the examples and documentation. but still i am stuck on this:

My RPDataExtraction.exe.config
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<!-- IMPORTANT: All keys are required unless they are noted as optional -->
<appSettings>
<add key="WSSPath" value="WSS"/> <!-- This is the path to the WSS logs -->
<add key="IISPath" value="IIS"/> <!-- This is the path to the ISS logs -->
<add key="LogParserPath" value="C:\Program Files\Log Parser 2.2\LogParser.exe" /> <!-- This is the path to the log  Parser application -->
<add key="DatabaseServer" value="localhost" /> <!-- This is the ip address or hostname of the SQL server used for reporting -->
<add key="StagingDatabase" value="dbSPSReportingStaging" /> <!-- This is the name of the Staging Database -->
<add key="ReportingDatabase" value="dbSPSReporting" /> <!-- This is the name of the Reporting Database -->
<add key="SearchAgent" value="MS Search 4.0 Robot" /> <!-- This is the search term used to filter search crawl hits from the WSS logs -->
<add key="Property1" value="Title" /> <!-- This is the key of the first property to extract from the site owner's user profile -->
<add key="Property2" value="WorkEmail" /> <!-- This is the key of the second property to extract from the site owner's user profile -->
<add key="Property3" value="WorkPhone" /> <!-- This is the key of the third property to extract from the site owner's user profile -->
<add key="MainPortalURL" value="http://localhost/" /> <!-- This is the url to the portal site (optional, used to get user profile data for wss sites) -->
<add key="PauseOnError" value="false" /> <!-- This tells the application whether to pause before closing on an application error (optional) -->
<add key="PauseOnComplete" value="false" /> <!-- This tells the application whether to pause before closing (optional) -->
<add key="LogFilePath" value="RPDataExtraction.log" /> <!-- This is the path to the application's log file  (optional) -->
</appSettings>
</configuration>

the copylogs customized batch file:
ECHO D | XCOPY /S /Y /O /K C:\WINDOWS\system32\LogFiles\W3SVC1 IIS\HOUSTON\W3SVC1
ECHO D | XCOPY /S /Y /O /K C:\WINDOWS\system32\LogFiles\STS WSS\HOUSTON

my copylogs result from cmd.exe:
C:\Program Files\Report Packs\SPS\Data Extraction>copylogs_houston.bat
Does IIS\HOUSTON\W3SVC1 specify a file name or directory name on the target
(F = file, D = directory)? D
C:\WINDOWS\system32\LogFiles\W3SVC1\ex051208.log
1 File(s) copied
Does WSS\HOUSTON specify a file name or directory name on the target
(F = file, D = directory)? D
C:\WINDOWS\system32\LogFiles\STS\FB44DB2F-51AF-40E3-96E2-22FCE87DE291\2005-12-08\17.log
1 File(s) copied

C:\Program Files\Report Packs\SPS\Data Extraction>RPDataExtraction.exe
Parsing IIS Log files...

Parsing C:\Program Files\Report Packs\SPS\Data Extraction\IIS\HOUSTON\W3SVC1 with Log Parser...finished

Parsing WSS Log files...

Parsing C:\Program Files\Report Packs\SPS\Data Extraction\WSS\HOUSTON...
An exception has occured:
System.UnauthorizedAccessException
Access to the path "C:\Program Files\Report Packs\SPS\Data Extraction\WSS\HOUSTON" is denied.

the error in bold is where i am at. I did the whole "everyone" has all permissions on the folder (non-production sandbox) and that didnt change anything either.

something strange is that if i put the C:\WINDOWS\system32\LogFiles\STS\FB44DB2F-51AF-40E3-96E2-22FCE87DE291\2005-12-08\17.log file at the root of the <add key="WSSPath" value="WSS"/> it parses the file and works (that is.. if there are no calls to the central admin website which i believe is broken but no one can confirm this).

whats funny is i havent even gotten far enough to have that "well documented" bug.

so.... houston, we have a problem!

 
Answer #17    Answered By: Sharonda Mcfarland     Answered On: Apr 03

The /O switch on XCOPY will copy the ACL from the source to the target. Did you grant Everyone on the System32 subdir? (Or, just remove the /O switch.)

 
Answer #18    Answered By: Cory Brooks     Answered On: Apr 03

after the copy i relaxed the permissions, but the STS_WPG had read  control on it, which is all that should be needed, perhaps other than my account, which is local administrators.

strange eh?

 
Answer #19    Answered By: Ruth George     Answered On: Apr 03

I didnt relax permissions on any OS folder (bad idea yes)

where i relaxed the permissions was on the destination copy target WSS (inside the program  files/sps/data extraction dir) which is all that the DEP should be caring about yes?

do you know your service pack levels and how the box you run the reports on is configured?

i have:
Medium Farm - 1 index/job, 2 search/web (NLB), 1 sql cluster with 2 nodes in A/P.

index/job - win2k3 sp1 wss/sps sp2
web/search - same
sql cluster - win2k3 sp1 with sql2k sp4 (havent even gotten to Reporting services part yet, refuse to install it until i can cleanly get the data into the database)

 
Answer #20    Answered By: Peter Peterson     Answered On: Apr 03

I don't use the /O switch, so the permissions from the OS are not copied.

The STS_WPG group is not involved, rather the user account that is running the DEP. And that account needs access to the SQL Server box, the SharePoint portal and the file  system..

I have it installed on one of the front-end servers and a database server (not the SPS database server).

 
Answer #21    Answered By: Kalyan Pujari     Answered On: Apr 03

Fred's white paper should be out on Mindsharp's Premium Content site. If
you don't find it please let me know (no Internet access right now).

 
Answer #22    Answered By: Laura Walker     Answered On: Apr 03

its not there........................

 
Answer #23    Answered By: Gopal Jamakhandi     Answered On: Apr 03

It was written over a year ago. I've pinged Fred and when I find the
article I'll make sure it is posted again.

 
Answer #24    Answered By: Kristina Cox     Answered On: Apr 03

i might have discovered something..

in my development environment the dep utility  works just fine but in my
staging/production environment it does not. the reason for this i am deducting
is because it is recieving different Group Policy settings. more tomorrow after
i apply my production environments' GPO settings to my.

 
Didn't find what you were looking for? Find more on SharePoint Usage Analysis Or get search suggestion and latest updates.




Tagged: