In an earlier post, I summarized the business benefit and overall performance improvement we achieved in a data conversion program that loaded data into a new Microsoft CRM system. I have received several questions asking for more detail on the problem and the techniques used. This post provides some additional detail. (more…)
October 15, 2011
May 31, 2011
How to speed up a data conversion program that loads Microsoft CRM 4.0
Overview
Our organization recently adopted Microsoft CRM 4.0. We developed a data conversion program to convert the data from the source systems and insert it into the new CRM system. Our initial effort was unsatisfactory because it ran way too slowly. We eventually solved our performance problems, and experienced a six-fold improvement in performance over our initial design! This report summarizes the challenges we faced, and our solutions, so that others who face performance issues with data conversion programs might benefit from the lessons we learned. Some of the lessons are specific to Microsoft CRM 4.0. But other lessons are of general interest. (more…)
How to avoid socket exhaustion on web servers by disabling HTTP Keep Alive
Problem:
Given
a web server, running IIS,
Hosting the web services for Microsoft CRM 4.0,
and a console application that runs a conversion program, which tries to update thousands of records per second through the CRM web service,
When
we run the conversion application at its fastest possible speed,
We observe
After only 26 seconds, we see failures in the web services calls of type System.Net.Sockets.SocketException.
If we put a deliberate delay of 100ms in the loop, we are able to process without errors, but at a reduced speed. (more…)
May 25, 2011
Parallel Programming with the Task Parallel Library from Microsoft
Last week I attended a VERY cool talk on some new ways to do parallel programming, using the Task Parallel Library (TPL) introduced in .NET 4.0. Microsoft has made it so easy, that even a frozen caveman programmer like me can do it! If you are trying to decide what to learn next month, put this near the top of your list. (more…)