The code for UDC (collector & analysis) is located at the following Subversion repository URL:

svn://svn.sharpdevelop.net/sharpdevelop/trunk/SharpDevelopServers/UsageDataCollector

The solution for the collector can be found in the /Project folder, appropriately named UsageDataCollector.sln. The structure of this solution looks like this:

Please note that under Reference Projects, there is a project reference to UsageDataCollector which actually lives inside the SharpDevelop client project structure. Therefore we recommend you check out the entire trunk when working with UDC.

Projects explained:

• CollectorServiceHost: a Web application that hosts service implementations from CollectorServiceLibrary. Really a thin wrapper only.
  
• CollectorServiceLibrary: the "meat of UDC" so to speak. It contains service implementations (message upload), import functions, msbuild tasks (wrapping the import functions) as well as some utility classes.
  
• CollectoryLibrary.Tests: way too few tests.
  
• CollectorServiceTestClient: a sample client that shows how client & server talk with each other.
  
• BulkImport: a command line client to import UDC messages. It is already superseded by the msbuild tasks because those are easier to extend.
  
• ImportWindowsService: a placeholder for a Windows service that automatically loads messages on arrival.
  
• DataAccess: all EF magic ist placed here.
  
• QuickViewer: a simple drag & drop viewer for UDC messages. A helper utility for various stages of development.

I skipped over the ExcelReport project for one reason: it was the initial PoC for data analysis (exceptions). The code is now part of the Web frontend we are building - but more on the analysis Web application in a later blog post.

In the last blog post, I talked about the initial signup message that is sent to our servers.Today, I want to shed some light on how the usage data is stored on the client.

The usage data - session information (time spent), environment information (OS, et cetera), feature use and exceptions (if any) is stored in a local SQLite database, which is located in the user's roaming profile folder at the following location:

C:\Users\<username>\AppData\Roaming\ICSharpCode\SharpDevelop4.0

The file itself is named usageData.dat. Although the extension is non-standard for SQLite, you can use one of the various SQLite management tools to have a look inside at the information (I am using SQLite3 Management Studio here):

Point it to the database file (please note that the file doesn't exist if you didn't opt in for UDC):

Now you can peruse the tables & information that is stored by UDC (please note that no information will be in the tables once UDC has uploaded sessions - once data for a session has been uploaded successfully, the associated data is purged locally):

The tables are: Sessions, Environment, Exceptions, FeatureUses as well as Properties. The latter is used to store the client id (see earlier blog post):

That's it - no magic proprietary file format, easy for you to peruse.

Good morning - today is the day: we have reached five million downloads on SourceForge! (the figure does not include downloads from CodePlex, our build server or third-party hosting sites)

Congratulations to the team for their hard work over the past nine years and seven months!

This is the start of a series of blog posts describing the Usage Data Collector that is built into SharpDevelop 4.0. UDC as it is called for short is intended to help us better understand what features are being used, how developers found out about the feature (menu, toolbar, context menu, et cetera), how much time they spent using the feature, and if there were any issues (exceptions). The intention is to find out which features need to be surfaced differently so more people use them, which the popular features are (and thus should be given a higher priority) and a lot more.

Before diving into the topic of this blog post, one important piece of architecture information: UDC is not tied to SharpDevelop. We developed it to be reuseable in your applications. Including (a yet to be developed) analysis Web frontend. That's also why we intend to give you a good background on what we collect, how we collect it, how it is transferred - so you have a good understanding whether it is suitable for your userbase.

On its new start page, SharpDevelop 4.0 asks you to participate in the customer experience improvement program:

If a user wants to participate and clicks the "Save" button, we do send the initial usage data message - for two reasons: first, depending on firewall settings, the user might need to allow SharpDevelop to communicate with the Internet. Doing it right now and there allows the user to see why SharpDevelop wants Internet access. Secondly, we gather Session "0" - information about the version of SharpDevelop, and on top of what platform it is running.

In terms of actual data, this is a sample UDC signup message (line 1 has been truncated for xmlns elements):

 1:  <UsageDataMessage ...>

 2:    <Sessions>

 3:      <UsageDataSession>

 4:        <EndTime i:nil="true" />

 5:        <EnvironmentProperties>

 6:          <UsageDataEnvironmentProperty>

 7:            <Name>platform</Name>

 8:            <Value>Win32NT</Value>

 9:          </UsageDataEnvironmentProperty>

10:          <UsageDataEnvironmentProperty>

11:            <Name>osVersion</Name>

12:            <Value>5.1.2600.196608</Value>

13:          </UsageDataEnvironmentProperty>

14:          <UsageDataEnvironmentProperty>

15:            <Name>processorCount</Name>

16:            <Value>1</Value>

17:          </UsageDataEnvironmentProperty>

18:          <UsageDataEnvironmentProperty>

19:            <Name>dotnetRuntime</Name>

20:            <Value>4.0.30128.1</Value>

21:          </UsageDataEnvironmentProperty>

22:          <UsageDataEnvironmentProperty>

23:            <Name>appVersion</Name>

24:            <Value>4.0.0.5623</Value>

25:          </UsageDataEnvironmentProperty>

26:          <UsageDataEnvironmentProperty>

27:            <Name>language</Name>

28:            <Value>es</Value>

29:          </UsageDataEnvironmentProperty>

30:          <UsageDataEnvironmentProperty>

31:            <Name>culture</Name>

32:            <Value>es-ES</Value>

33:          </UsageDataEnvironmentProperty>

34:          <UsageDataEnvironmentProperty>

35:            <Name>userAddInCount</Name>

36:            <Value>0</Value>

37:          </UsageDataEnvironmentProperty>

38:          <UsageDataEnvironmentProperty>

39:            <Name>architecture</Name>

40:            <Value>x86</Value>

41:          </UsageDataEnvironmentProperty>

42:        </EnvironmentProperties>

43:        <Exceptions />

44:        <FeatureUses />

45:        <SessionID>0</SessionID>

46:        <StartTime>2010-03-18T12:01:38.5625Z</StartTime>

47:      </UsageDataSession>

48:    </Sessions>

49:    <UserID>cb02e999-4e80-45d2-9773-ed3d80b3937f</UserID>

50:  </UsageDataMessage>

Please note line #49: each client machine creates a GUID for itself. This GUID is used to map all sessions to a single user - so we can tell whether a single developer is using a feature heavily compared to say a hundred developers using the very same feature lightly.

As you can see, the initial message is very small. Once data is being gathered, the messages do increase in size, and contain more information (feature use, exceptions). We will look into those message in future blog posts, as well as how those messages are processed and analyzed.

With the release of RC1, the code converter site has been upgraded too (from NRefactory 3.2.0.5362 to 3.2.0.5506). Relevant checkins:

http://fisheye2.atlassian.com/changelog/sharpdevelop/?cs=5432
http://fisheye2.atlassian.com/changelog/sharpdevelop/?cs=5448

Supported languages (via Web interface as well as Web Service): C#, VB.NET, Boo, IronPython, IronRuby.

This blog post announces two major changes for SharpDevelop 3.2: a setup size reduction from 21.1MB down to 14.6MB (a clean 30% less than before), and an upgrade of Sandcastle Help File Builder support to version 1.8.0.3.

So how did we shrink the setup? Let's look at some sizes first:

Revision 5493 is before the upgrade of SHFB support (happened in 5499), and 5500 is the latest setup build - with additional tweaks. So what are the tweaks?

Basically, we had a look at feature usage. Like "How many people actually know that we are shipping the WiX *.chm documentation with SharpDevelop?" or "How many people actually generate documentation chm files from XML comments?". Turns out that the answer is "not too many".

The delta of 5499 to 5500 shows the effect of the WiX help documentation. And 5493 to 5499 the delta of the old release of SHFB.

Which brings me to the upgraded functionality: we had a 1.7 release in the box, and 1.8.0.3 uses a different project file format. We had to upgrade that for "Build documentation..." (project context menu) to work properly with the new release.

If you don't have SHFB on your computer (which is now the default after installing SharpDevelop), you will see the following dialog:

This dialog is now standardized (we haven't installed TortoiseSVN in the past either). Simply follow the link, install the latest version, and you are good to go (even without restarting SharpDevelop, we perform the "SHFB installed?" check on the fly):

Note that we produce a valid .shfbproj file with a reference to the project you just had open.

So why are we shipping some stuff in the box and some not? That mostly depends on how deeply integrated a feature is in the IDE (well, after all that is short for integrated development environment). We have to have control over certain tools and the version being used. In some cases we can leave it up to the user to install it if needed, like F# support, Subversion support, FxCop or StyleCop support, or now documentation generation.

Lean & mean is the new green.

SharpDevelop wraps F# Interactive (fsi.exe) in a pad of its own. For SharpDevelop 3.2, it looks like this:

You type the code you want to evaluate in a single-line textbox on top of the fsi.exe output.

With revision 5498 (part of SharpDevelop 4.0), Daniel reused Siegfried's approach for the debugger console to make F# Interactive a lot more compelling:

You type right inside REPL. Way more useful!

Boot note: In revision 5494, we ported the F# backend binding from F# (yes, the F# binding was written in F#) to C#. The main reason was that requiring F# - which is not part of the basic .NET installation - caused us problems (again) with the automated build.