John Nye

Agile software developer

My name is John Nye. I am a developer from Essex, working in London. I specialize in C#, ASP.NET MVC, Test driven development (TDD), Entity Framework, HTML5, Javascript and JQuery. I have built this blog from scratch using ASP.NET MVC 4, Entity Framework code first and markdowndeep.

If you have any feedback on my blog or just want to strike up a conversation I'd love to hear from you. You can get in touch with me via twitter @ninjanye or by emailing using the links in the header.

Check out my latest posts below or use my tag cloud for previous posts on a specific subjects

Improved Levenshtein searching with SearchExtensions

NOTE: This post will not cover what Levenshtein Distance is or how it is calculated.
For more information on Levenshtein Distance, please visit the Wikipedia page

Up until now, NinjaNye.SearchExtensions has only been able to calculate the Levenshtein distance between:

  • A single property and a string search term
  • A single property and one other property

To celebrate 10,000 downloads of SearchExtensions, the latest version allows you to calculate the distance between:

  • A single property and multiple strings
  • Multiple properties and a single string
  • Multiple properties and multiple strings
  • A single property and multiple other properties
  • Multiple properties and a single property
  • Multiple properties and multiple other properties

You can install NinjaNye.SearchExtensions from nuget using the following:

PM> Install-Package NinjaNye.SearchExtensions

All of this is fairly difficult to describe, so below are some examples to help me explain.

Calculate levenshtein distance against multiple terms

The following functionality has been in SearchExtensions for a little while but it is a good starting point to describe how to get the Levenshtein distance between each users first name and the word "Jim"

var result = context.Users.LevenshteinDistanceOf(x => x.FirstName)
      .ComparedTo("Jim");

The above will return each user along with the Levenshtein distance of the FirstName compared to "Jim". In version 2.0, it is now possible to compare the FirstName to more than one term:

var result = context.Users.LevenshteinDistanceOf(x => x.FirstName)
      .ComparedTo("Jim", "Fred");

Calculate levenshtein distance against multiple properties

Calculating the distance against multiple properties is also now supported:

var result = context.Users.LevenshteinDistanceOf(x => x.FirstName, x => LastName)
      .ComparedTo("Jim", "Fred");

The above will result in 4 comparisons per record. Because the levenshtein calculation must be performed in memory, you should always reduce the record set as much as possible beforehand, e.g:

var result = context.Users.Search(x => x.CountryCode)
      .EqualTo("GB")
      .LevenshteinDistanceOf(x => x.FirstName, x => LastName)
      .ComparedTo("Jim", "Fred");

When performing a Levenshtein search it is important to always reduce your record set as much as possible, especially when comparing multiple properties to multiple terms

A new result

The result of a Levenshtein search also changed and now has additional properties to work with:

public interface ILevenshteinDistance<out T>
{
    /// <summary>
    /// The distance of the first comparison
    /// </summary>
    int Distance { get; }

    /// <summary>
    /// The queried item
    /// </summary>
    T Item { get; }

    /// <summary>
    /// A collection of all distances calculated
    /// </summary>
    int[] Distances { get; }

    /// <summary>
    /// The minimum distance of all levenshtein calculations
    /// </summary>
    int MinimumDistance { get; }

    /// <summary>
    /// The maximum distance of all levenshtein calculations
    /// </summary>
    int MaximumDistance { get; }
  }

This means you can order the results so that the record with the closest match is ordered first:

var result = context.Users.LevenshteinDistanceOf(x => x.FirstName)
      .ComparedTo("Jim", "Fred")
      .OrderBy(x => x.MinimumDistance)
      .ThenBy(x => x.MaximumDistance);

If you would like to know more about this or any other feature, please get in touch by adding a comment below, contact me on twitter (@ninjanye) or you can raise an issue on the github project

I woke up this morning to find that search extensions now has over 10,000 downloads!!!

I wanted to take this opportunity to thank all of you that have contributed to the project be it by simply using SearchExtensions, requesting new features, raising issues on GitHub or even submitting PRs.

The future

The improvements I'd like to make to SearchExtensions include:

  • Support for .Net Core
  • Split out Soundex support to its own package
  • Split Levenschtein support to its own package

The above are only what I see as a way to improve SearchExtensions. The best, and most useful ideas often come from users. With that in mind:

If you have any suggestions on how the package could be improved, please get in touch by adding a comment below, contact me on twitter (@ninjanye) or you can raise an issue on the GitHub

PM> Install-Package NinjaNye.SearchExtensions

SearchExtensions now supports searching of child collections for IQueryable

You can now perform child collection searches on IQueryable objects which will get translated and run on the data source server returning only the filtered records.

String searching

The functionality allows for the following to be performed on the data provider and not in memory

var shops = context.Shops.SearchChildren(s => s.Products)
                         .With(p => p.Name)
                         .Containing("ball");

The above returns shops that contain any product that has the term "ball" in it's Name field.

The above will send the following SQL when using LinqToEntities

SELECT [Extent1].[Id] AS [Id]
  -- Additional columns --        
FROM [dbo].[Shops] AS [Extent1]
WHERE EXISTS (
  SELECT 1 AS [C1]
  FROM [dbo].[Products] AS [Extent2]
  WHERE ([Extent1].[Id] = [Extent2].[Shop_Id])
    AND ([Extent2].[Name] LIKE N'%ball%')
)

Support is also in place for searching multiple properties and multiple values

var shops = context.Shops.SearchChildren(s => s.Products)
                         .With(p => p.Name, p => p.Desc)
                         .Containing("ball", "balls");

Additional string comparisons

All the usual string comparisons that you expect from SearchExtensions are also supported for searching child records:

  • .StartsWith() - property starts with any of the given terms
  • .EndsWith() - property ends with any of the given terms
  • .EqualTo() - property is equal to any of the given terms
  • .Containing() - property contains any of the given terms
  • .ContainingAll() - property contains all of the given terms

Support for non string filtering

Support has also been added to allow searching child collections for matches on other types. The following example returns shops who have products with a price between 10,000 and 100,000:

var shops = context.Shops.SearchChildren(s => s.Products)
                         .With(p => p.Price, p => p.RRP)
                         .Between(10000, 100000);

Both the examples so far will return a collection of Shops where any of it's products match the given criteria.

The equivalent of the above without utilising SearchExtensions is as follows:

var shops = context.Shops.Where(s => s.Products.Any(p =>
                              (p.Price > 10000 && p.Price < 100000)
                           || (p.RRP > 10000 && p.RRP < 100000)));

All the usual non string comparisons have also been implemented for IQueryable searching:

  • .GreaterThan()
  • .GreaterThanOrEqualTo()
  • .LessThan()
  • .LessThanOrEqualTo()
  • .Between()

Method Chaining

As is the norm with SearchExtensions, It is also possible to chain your search criteria together in order to create more complex filtering:

var result = context.Countries.SearchChildren(x => x.Cities)
                    .With(c => Name, c => c.LocalName)
                    .StartsWith("A", "E", "I", "O", "U")
                    .EndsWith("S", "T", "U", "V", "W");

To read more about SearchExtensions, please visit ninjanye.github.io/SearchExtensions/


Thanks again to @bzbetty for getting in touch via a github issues and creating this feature request

If you have a new feature request, please get in touch by adding a comment below, contact me on twitter (@ninjanye) or you can raise an issue on the github project page

To install SearchExtensions, you can either install it via the Nuget Package manager or by using the following in the package manager console

PM> Install-Package NinjaNye.SearchExtensions