Steady as she goes

Its been a particularity normal week. I am continually going through mounds and mounds of documentation just to find information on the simplest things. For example, the supported events are very poorly documented. No one source lists all the events and what causes each event to trigger. I plan to fix that in the coming weeks. Despite this I was able to get the sensor tool to record some of the larger types of data that are strictly (I hope to go beyond the event system) given by VSTFS’ event system:

• Additional Source Control changes (Branching, moving)
• Build Systems
• Work Items

There were/are also some connection issues when it comes to the sensorshell (the little guy that sends all the data to the hackystat server). I was able to restructure the web app so that only one instance of the sensorshell was created at a time 🙂 . This means that data can be quickly sent to the server or stored offline without the need to restart the service on every event. The problem with this approach though is that if the server crashes or is forcefully reset/stopped without making sure all sessions have been closed properly the child process (sensorshell) gets abandoned and outputs a text file like this.

… forever!!! Did I mention that it also takes up 100% of the CPU. For now this means that every time I run the development version I must forcefully kill the sensorshell process. I believe it has something to do with the instantiation of the sensorshell. I am currently using some of the code from the Visual Studio Sensor to initiate the shell and I may look into how it actually does this to see if this could possibly be the problem. It may also have something to do with multisensorshell. I will have to pour over the Hackystat docs to see exactly how this works.

Some other small fixes. I fixed the logging so that it now outputs to daily files and appends to the end of each log with a nice timestamp. Its a simple fix but it really helps. I finished the settings module. It will not look for the VSTFS server, create the Hackystat database/catalog if not already created, create the settings table and store the appropriate settings. Useful for anyone who wants to start testing this sensor right away. I have also made some minor changes to the wiki page, given myself a load of tickets and overhauled the subversion repository.

Plan of attack for next week is to continue implementing event detectors and begin writing some unit tests. I foresee problems with the whole unit-test situation as there are so many independent systems that if even one of them errors then the test will fail.

Advertisement

Source and Wiki

A new url for everyone who wants to keep up-to-date:

Hackystat Wiki and SVN

The SVN should be readable so people can checkout a read-only copy and hopefully compile for there own use.

Busy week

It’s been a very busy week. Aside from my own personal life I was able to get some data sent all the way from VSTFS to the development Hackystat database we have setup at UofT. It uses the pre-existing sensorshell to send data which gives the benefit of offline storage and product longetivity with a con of being slower than just making my own. Production has been good and the sensor now sends alot more data about version control than is shown in the screenshot below. I hope to utilize as much of the data as is provided by VSTFS event system to send to Hackystat and maybe even some that isn’t 😉 . Right now it easily sends the equivalent amount of information as the SVN sensor. I shall provide screenshots next week.

In case you were still wondering what VSTFS is exactly, its basically some super-powered team management software and includes all of the tools needed to facilitate team coordination and planning. Its like the corporate edition of DrProject. Right now the tool senses version control info but has the ability to sense more I just need to decide what I want to send and what is overhead info.

One of the most frustrating things about developing a sensor for such a large and requirements dependent system is the setup. Thats why I plan to make the setup and configuration of this sensor as easy and foolproof as possible. On that note, I am done about 70% of the initial settings infrastructure interface that will allow users of this very large sensor to quickly and easily configure critical sensor properties and store them in the (required) running MSSQL Server that stores the other various VSTFS information. Since all settings need to be stored in an sql database there would need to be some heavy security authentication going on if the user wishes to access the sensor properties outside the local system. Therefore I decided that at the moment it can only be run and changed from the computer taht is running VSTFS. I hope to change that in the future… sometime… maybe… we will see.

I am finally at that point in the project where you (and possibly your team) have built up a codebase large enough that you can actually see components working together and really begin to see how the system is functioning and what needs improvement. Its been a bit slow and I’ve had to brush up on my SQL (especially for database creation) as well as SOAP and XML. With a good foundation now its hard to choose from the tons of cool ideas I have on how to extract more data and how to run things more efficiently. Overall its proving to be a really good learning experience.

First data sent!

Today the service sent the first data ever. The data was very simple but its still cool that it works.

Don’t get to happy yet though. The code is all over the place and a ton of features, methods, properties, functions, etc are missing or need heavy refactoring. It definitely helped that there was already a wrapper class for the visual studio plugin that I could use. It may be possible to build further upon the existing code to make it easier for other people developing sensors in c#.

Isn’t it cool how sometimes things just work.

Today everything was just working, and working well. The virtual machine was running super-fast, never glitched. My internet was fast and consistent (for once). I was able to quickly solve old problems I had with the VSTFS sensor and start making some real progress. The blogging software had all the nifty features that made posting and editing easy instead of cumbersome. Considering the number of mishmashed systems and application I am using simultaneous it just makes developing wonderful when it all works together.

Woot for progress!

So today I finished the subscription part of the application. I wrote a small app that would determine what events were available (unfortunately those are manually imported into an xml ATM), displays which events are currently be registered by querying the database, and also allows for manipulation of registred events (subscribing/unsubscribing) using checkboxes. The interface is mediocre but the whole system works very nicely and can be used to easily get the VSTFServer sending the specified events to the services defined in the TFS Notification Web Services Template. There is a ton of testing and a few hard-coded items that must be changed later; but for now step 3 is basically complete.

More of a personal reminder to myself than anything to report. It may be worth while looking at the possibility of using polling to extract other information that is not reported using a webservice by VSTFS. It would be slower but that database has a TON of info in it and its only just waiting for me to extract it. (BTW: SQL is teh awesome!)

The next step (goal for next week) is to build a simple hackystat connecter that will allow for data collected through the webservice to be sent to a hackystat sensorbase. This may be a bit trickier because the various events all collect different types of data so writing a unified general adapter may be a bit tricky if not impossible.

So many problems

Yesterday there was a massive bug in the eclipse plugin for Hackystat. Somehow it managed to record over 30GBs of data!!! Unfortunately this crashed the server which only had a 50GB drive most of which was filled with a large virtual machine and the client using the plugin who also had a small drive which ran out of space bringing their system to a halt. The problem has been fixed and the Hackystat team has been made aware but its a little discouraging. Hopefully during the course of this GSoC project I won’t encounter too many more bugs :p .

In other news I have begun work on the actual sensor and am currently in the midsts of writing a web-interface to allow for some basic sensor configuration and VSTFS connection items before I actually write the code to connect to the hackystat server. I am doing a ton of re-learning since I last used C# and ASP about 8 months ago. Overall though its pretty cool and I can’t wait till the first pieces of data start flying towards the Development Hackystat server.

P.S. When developing open-source software the key to reeling in the vast majority of clients is to develop software that is easy to install/update/repair/re-install/remove. Very few “average Joe’s” want to build your system from source, configure the environment variables, change the settings files, install dependancies… etc, etc, etc. As you can tell I have done way to much installing/repairing recently.

The current plan

So if anyone is wondering what exactly is going through my head at the moment, don’t worry, I have a fool proof plan…

<Step #> <Codename>: Description

1. Alpha: Setup the environment. (Complete)
2. Two’s company: Basic research into the architecture and design of a VSTFS app. (Complete)
3. Three’s a Crowd: Design a simple app that can listen for a small subset of the available events triggered by TFS.
4. The Fantastic Fourth: Design a simple app that can send data to a Hackystat server. May include looking at current visual studio sensor for Hackystat.
5. 5th element: Put simple apps together to get a beta app for the mid-point of GSoC.
6. Sixth Sense: Advanced research into the more obscure events/information I can get from VSTFS to send to Hackystat.
7. Lucky seven: Build on the simple apps to increase functionality.
8. Because 7,8,9: Begin work on some unit-tests and the documentation as a Google code-project.
9. Cats: Create an easy to use installer.
10. Decimal: Bug-testing.
11. Hour: Final clean-up and code submission.
12. Hex: Sit back and relax).

So thats the general idea. Obviously some parts may take longer than others but hopefully nothing will be too simple or too challenging.

With progress there is change!

So yesterday and today I delved into samples from the Visual Studio Team Foundation Server 2005 SDK which is a subset of the Visual Studio 2005 SDK. My goal was to investigate some way of listening to TFS (Team foundation server) events as outlined in the documentation I had been reading during the massive installation I had been working on. I learned two things (one of which I had already suspected since the beginning):

1. The “Team Foundation Server” does not directly allow for plugins/add-ins to be added (from what I have read and seen). I suspected this already and therefore prepared myself for some client<->server programming.
2. The existing architecture for listening to Events is web-based, which is still client-server and involves me programming in c# but may require some additional setup for whomever wishes to install the sensor in the near future.

I didn’t expect number 2 but after about an hour of searching around the web I found a nifty bit of code template to confirm my theory.

• TFS Notification Web Services Template

It took me awhile to get it running as I had never used anything but the default empty template before but after some more poking around the web and fiddling with using BisSubscribe to tell the server I was listening for events I was able to get this:

So now that a have a starting point for next week that will focus on building a small web-app that listens to some events and has some customizability. Overall its looking pretty good so far.

Microsoft Does Development Right

If there is one thing Microsoft does right its development tools. After using Eclipse, various python editors, Dev C++, Vi, emacs, etc… using Visual Studio again is like developing using “big-boy” tools. The interface is fairly simple and its very easy to find things. The debugger is easy to customize and learn. It can generate simple class diagrams from your source, can easily be linked to source control. Overall it just feels as though the environment was designed to help me not to be endlessly fiddled with like other IDE’s. It may not be open-source but it is extendible.

When using Visual Studio there is just a sense of stability (ya! thats right). Eclipse, Dev C++ crashed so often for me and often brand my computer to its knees. I’m currently running Visual Studio in a virtual machine and although its slower than normal its much faster than eclipse.

The only downside to using Visual Studio is that the only programs you can write with it that are (easily) portable to other systems besides Windows are c++ apps. Unless I am missing something really obvious.