This project is read-only.

Wonderful improvement to LINQ

Dec 3, 2009 at 10:18 AM

I just want to congratulate you guys on nice work. I have passed through nightmare performance issues with linq. I working on an old database schema and had to import everything to memory in order to export to new schema. I'm talking about hundreds of thousands of records.

I took the hard path after discovering the LINQ weakness and used optimized Hash Tables to make it fast. I think we had same objectives but sure your work is far more reusable.

So how can I help ...

Dec 13, 2009 at 3:15 AM

Sorry to put this back on you... but how do you think you can help?

We love new ideas, patches, or anything you can think of.

Dec 15, 2009 at 10:09 AM

Well it looks promising for indexes based on string values. unfortunately it didn't help me on integer based indexes which I assumed it would help. Somehow the join and the where processing should be reworked !

Dec 15, 2009 at 1:56 PM

I do agree, that there is work to be done with the Linq expression handlers. Would love to apply any improvements you can find.

Given the following benchmark, can you make this fail? (It doesn't fail for me) 

Here's the typical console output i see:

           indexed=[00:00:00.0428251], non-indexed=[00:00:00.3005877]

 

[Test]
public void Should_work_with_integer_indexes()
{
	var random = new Random();
	List<SimpleClass> items = Enumerable
		.Range(0, 10000000)
		.Select(s => SimpleClass.New(random))
		.ToList();

	var nonIndexedStopwatch = new Stopwatch();
	var indexedStopwatch = new Stopwatch();

	nonIndexedStopwatch.Start();
	var allItemsFoundWithoutIndex = items.Where(w => w.SomeIntValue == 10).ToList();
	nonIndexedStopwatch.Stop();

	var spec = new IndexSpecification<SimpleClass>().Add(a => a.SomeIntValue);
	var indexedItems = items.ToIndexableCollection(spec);

	indexedStopwatch.Start();
	var allItemsFoundWithIndex = indexedItems.Where(w => w.SomeIntValue == 10).ToList();
	indexedStopwatch.Stop();

	Console.WriteLine("indexed=[{0}], non-indexed=[{1}]", indexedStopwatch.Elapsed, nonIndexedStopwatch.Elapsed);

	Assert.Less(indexedStopwatch.Elapsed, nonIndexedStopwatch.Elapsed);
}