Fluent-ish Validations – Part 2 [Solution]

Part 1 is here

I started looking at the Validator source code to see how it works internally. If you didn’t know ASP.NET MVC and Entity Framework uses the Validator to perform their validations. The Validator uses the TypeDescriptor that’s good news as we can add attributes to objects and types at runtime with the TypeDescriptor.

Side Note: I was getting excited thinking about being able to add validation to specific object and not their type. For conditional validations but the Validator will always look at the object type to get the attributes and not the object itself.

Even though this is mainly about Validation in the end we are just adding attributes so I decided to not just called it Validation and name most things with the prefix of Attributes. Really once we are done we could add any attribute to a property. To start we are going to need to create a CustomTypeDescriptor.

To give a shout out I got the gist from this stackoverflow answer but I implemented it a bit different to make it more generic to what I want to do.

To make it simple we will take in the constructor an Enumerable of attributes to add to the class level and an Enumerable of string/attributes to add to properties. I called this class AttributeTypeDescriptor. Need to add a TypeDescriptionProvider to make sure of this CustomeTypeDescriptor. Nothing to fancy just tell the TypeDescriptor about it and then have a method to add an Attribute to the class and one to add an Attribute to a property. I called this class AttributeProvider.

Here’s where we start to get into the fluent-ish part. I wanted an easy way to configure attributes so I made an interface called IAttributeConfiguration. This interface allows adding property level attributes either by expression or by string if using nameof() and attributes to the class itself. Since it’s generic it already knows what type to add them to. I created the implementation of this called AttributeConfiguration which has a default constructor and one that allows you to replace the provider if you’d like.

Now that we are adding attributes at runtime we are no longer restricted by things needs to be constants when declaring the Attribute. Following the IValidatableObject pattern I was using where each validation was it’s own method I’m going to create a ValidationAttribute that will take a Func. Called this class ValidatableObjectAttribute. Now I need to create a class to handle adding property validation and class validations. Going to skip over some of the implementation details here but is the interfaces/class responsible IPropertyValidations, PropertyValidations, IObjectValidations, ObjectValidations and some extension methods to make accessing them easier.

One of the common things I run into with validations is skipping validation if another property isn’t valid. With that in mind I create a new Validation Attribute called ValidateIfAttribute. This attributes takes a dictionary that the key is the string name of the property and a func of how to get the value of the property and the ValidationAttribute that should be executed if all the other properties are valid.

Word of warning with this you can create a cyclical validations. For Example if FirstName is requiring LastName valid and LastName is requiring FirstName valid. But as a developer I would prefer to have the power with knowledge of the short comings and deal with it then not have the option at all.

In the Property validation there is a setter for if you need to set some properties for the validation attribute that is being added but you also noticed that the setter needs to return an attribute. For the most part returning the attribute that came in back out is what you will want to do but I had it return the attribute for this case. I want to wrap the incoming attribute in the ValidateIfAttribute so that the incoming attribute will only fire if the other properties are valid.

I create an extension method called WhenValid that would return the Func to use to wrap the setter method in. By passing in an Expression in we can get both the name of the property and compile the expression to get the func needed to access the property. I still feel this part is a bit awkward at the moment and would like to see if it can become more fluent.

Now lets go back to the original PO issue. With this solution we can create a class for the validation

public class SalesOrderLineItemsValidations
{
    private readonly IAttributeConfiguration<SalesOrderLineItems> _attributeConfiguration;
    private readonly IDataService _dataService;

    public SalesOrderLineItemsValidations(IAttributeConfiguration attributeConfiguration, IDataService dataService)
    {
        _attributeConfiguration = attributeConfiguration;
        _dataService = dataService;
    }

    public async Task Configure()
    {
        var itemValidations = _attributeConfiguration.ValidationsFor(x => x.Item);
        itemValidations.Add<RequiredAttribute>();

        var itemLength = await _dataService.ItemLengthAsync();
        itemValidations.Add(() => new MaxLengthAttribute(itemLength));

        var qtyValidations = _attributeConfiguration.ValidationsFor(x => x.Qty);
        var whenValid = _attributeConfiguration.WhenValid(x => x.Item);
        qtyValidations.Add(QtyGreaterThanZero, whenValid);
    }

    protected virtual ValidationResult QtyGreaterThanZero(decimal qty)
    {
        if (qty > 0)
        {
            return ValidationResult.Success;
        }

        return new ValidationResult("Quantity must be greater than zero");
    }
}

The Configure method is where the “Fluent-ish” validation happens at. Since I already have a BootStrap class and now I just need to have it add in the validation config classes and execute them.

Last bit of warning/thoughts.

The Validator does cache the attributes on a type the first time it uses the class. Which means if adding validation after the Validator has been called it will not pick up the new validations. Even if there is a way for them to know to refresh it *cough cough*. I don’t see this as a big deal as 99% of the time the validations would be added when the application starts up. I also thought that adding the validation in a different class then the original class might confuse someone coming after me but I feel that’s a training issue and frameworks like Entity Framework have their own configurations and they do allow adding some basic validations so it’s not unheard of. If I need to add a class validation I don’t know if it’s better to just add the IValidatableObject interface or keep all the validation in one class and have the fluent add the validation. I think that’s something the team should decide the pros and cons on.

All the code is hosted in GitHub and is licensed with the MIT license

Wednesday, December 6th, 2017 Validations No Comments

Fluent-ish Validations – Part One (The problem)

A little background. I write software that is then sold to VARs (Value-Added Resellers). They can take our code and enhancement it or change it based on their exact customer needs. We try to write code that allows them to change anything they might need. Now all software does have some basic data it needs to function properly but sometimes we put in ValidationAttribute to just make programming simple. Who doesn’t like ASP.NET MVC or Entity Framework to auto call our validation. But when you add an attribute to a class it’s really hard to take that attribute away or to change it. So typically I’d shy away from ValidationAttribute and instead implement the IValidatableObject Interface. But this leads sometimes to some ugly code.

Let take an arbitrarily contrived example. Maybe not too contrived but it’s simplified down from a real requirement.

User requirement: Create a screen for a user to buy items and needs the ability to enter the item, quantity and date requested.

Rules: The item needs to be filled in and the length of the item is configurable. Quantity must be positive. Date can be empty or any valid date.

Seems simple enough let’s create a class

public class SalesOrderLineItems
{
    public string Item { get; set; }
    public decimal Qty { get; set; }
    public DateTime RequestedBy { get; set; }
}

The business rule of item required is simple we can just slap on the RequiredAttribute but we can’t slap on the MaxLengthAttribute because the length is configurable. Ok we will just handle that with IValidatableObject interface.

To get the item’s length we have a DataService

public interface IDataService
{
    Task ItemLengthAsync();
    int ItemLength();
    Task QuantityPercisionAsync();
    int QuantityPercision();
}code>

Let's write up the validation for max length in the IValidatableObject


public class SalesOrderLineItems : IValidatableObject
{
    [Required]
    public string Item { get; set; }
    public decimal Qty { get; set; }
    public DateTime RequestedBy { get; set; }

    public IEnumerable Validate(ValidationContext validationContext)
    {
        yield return ValidateItem(validationContext);
    }

    /// This method just wraps the MaxLengthAttribute so we can set the length at runtime
    protected virtual ValidationResult ValidateItem(ValidationContext validationContext)
    {
        var dataService = (IDataService)validationContext.ServiceContainer.GetService(typeof(IDataService));
        var maxValidation = new MaxLengthAttribute(dataService.ItemLength());
        var context = new ValidationContext(this, validationContext.ServiceContainer, validationContext.Items)
        {
            MemberName = nameof(Item)
        };
        return maxValidation.GetValidationResult(Item , context);
    }
}

I'm a big fan of constructor injection but not for my models, they should have a default constructor. so we have to go back to using the Service Container (yuck) but it's possible. I also implement each validation in it's own method so the VARs can override a specific validation. I could have made it's own class and using the service locator grab the validation class and call those methods. In that class I could have injected the dataservice but still to get the validation class I need to use the Service Locator pattern. For this example we will leave as methods. For the Qty requirement let's slap on the RangeAttribute [Range(typeof(double),".01","9999999999999999")]

Now we demo to our Product Owner. First complaint is they don't like the range validation error message. Ok we can fix that with a Resource file to change the message. Second complaint is the number of decimals is configured and right now it's hard coded that .01 is the min. Product Owner likes it's trying to verify the minimum quantity that can be entered and not just greater than zero. Hmmm we were thinking of creating a MinAttribute but if they want it to be the min enterable amount based on the dataservice that is out. Product Owner also says they like that the quantity isn't validated if the item hasn't been filled in yet but....they would also like to skip the quantity validation if the item is too long because we know it's not valid. Ok we going to have to create another validation method.

Here's the code that will make our Product Owner happy

public class SalesOrderLineItems : IValidatableObject
{
    [Required]
    public string Item { get; set; }
    public decimal Qty { get; set; }
    public DateTime RequestedBy { get; set; }

    public IEnumerable Validate(ValidationContext validationContext)
    {
        var result =  ValidateItem(validationContext);
        // don't run any other validation if item didn't pass
        if (result == ValidationResult.Success)
        {
            yield return ValidateQty(validationContext);
        }
        else
        {
            yield return result;
        }
    }

    protected virtual ValidationResult ValidateItem(ValidationContext validationContext)
    {
        var dataService = (IDataService)validationContext.ServiceContainer.GetService(typeof(IDataService));
        var maxValidation = new MaxLengthAttribute(dataService.ItemLength());
        var context = new ValidationContext(this, validationContext.ServiceContainer, validationContext.Items)
        {
            MemberName = nameof(Qty)
        };
        return maxValidation.GetValidationResult(Qty, context);
    }

    protected virtual ValidationResult ValidateQty(ValidationContext validationContext)
    {
        var dataService = (IDataService)validationContext.ServiceContainer.GetService(typeof(IDataService));
        var percision = dataService.QuantityPercision();
        var minvalue = Convert.ToDecimal(1/Math.Pow(10, percision));
        if (Qty > minvalue)
        {
            return ValidationResult.Success;
        }
        return new ValidationResult("Quantity must be greater than zero");
    }
}

All that for a simple business requirements the code just keeps expanding if you have more requirements and can quickly get out of hand. We will look at Part 2 how to tame this beast.

Friday, October 27th, 2017 Validations No Comments

Combine Sliding and Absolute Expiration in MemoryCache

I wanted to use both absolute and sliding expiration for my cached item. By default the MemoryCache Class does not support this. I found a good answer on StackOverFlow Combine Sliding and Absolute Expiration but I wanted more flexibility than that solution so I looked at creating my own implementation of the ChangeMonitor.

Using the TPL I thought it would be nice to also support it with a CancellationToken as well. I reviewed the source for the SqlChangeMonitor and it seemed pretty straight forward.


    public class CancellationTokenChangeMonitor : ChangeMonitor
    {
        private readonly IList _disposers = new List();

        public CancellationTokenChangeMonitor(CancellationToken token)
        {
            UniqueId = Guid.NewGuid().ToString("N", CultureInfo.InvariantCulture);
            var shouldDispose = true;
            try
            {
                _disposers.Add(token.Register(() => OnChanged(null)));
                shouldDispose = false;
            }
            finally
            {
                InitializationComplete();
                if (shouldDispose)
                {
                    Dispose();
                }
            }
        }

        public CancellationTokenChangeMonitor(DateTimeOffset expiration)
        {
            UniqueId = Guid.NewGuid().ToString("N", CultureInfo.InvariantCulture);
            var shouldDispose = true;
            try
            {
                // if date is in the past then pass in zero
                var expires = new[] {expiration - DateTimeOffset.UtcNow, TimeSpan.Zero}.Max();
                var tokenSource = new CancellationTokenSource(expires);
                _disposers.Add(tokenSource.Token.Register(() => OnChanged(null)));
                _disposers.Add(tokenSource);
                shouldDispose = false;
            }
            finally
            {
                InitializationComplete();
                if (shouldDispose)
                {
                    Dispose();
                }
            }
        }

        protected override void Dispose(bool disposing)
        {
            if (disposing)
            {
                foreach (var disposer in _disposers)
                {
                    disposer?.Dispose();
                }
            }
        }

        public override string UniqueId { get; }
    }

Created two overloads for the constructor. One that will take an existing CancellationToken and the other a DateTimeOffSet, to set the absolute expiration date. Now I can sliding and absolute or tie it directly to a CancellationToken

Example:


    // Expire in 1 minute if no access
    //   or expire if cache is older than 1 hour
    MemoryCache.Default.Add("NHail", "Charles N Rice", new CacheItemPolicy()
    {
         SlidingExpiration = TimeSpan.FromMinutes(1),
         ChangeMonitors = {new CancellationTokenChangeMonitor(DateTimeOffset.UtcNow.AddHours(1))}
    });

Tags: ,

Wednesday, May 24th, 2017 Uncategorized No Comments

Move from WCF Data Services to Web API – Part 5

It’s been awhile since Part 4 and the project is now up on GitHub and it seems to be pretty stable. Hasn’t balked at a OData Query I sent at it yet. Since it’s based off Web API 2.1 it only supports V3 of the OData specification.

I changed ProcessRequest to return IHttpActionResult since that’s the Web API way.

protected virtual IHttpActionResult ProcessRequest()
{
    var pathHandler = Request.GetODataPathHandler();
    var metadata = GetMetadata();
    var path = pathHandler.Parse(metadata, GetODataPath());
    if (path != null)
    {
        Request.SetODataPath(path);
        Request.SetODataRouteName(ODataRoute);
        if (path.Segments.Count == 0)
        {
            // Requested service document
            return GenerateServiceDocument();
        }

        if (path.Segments.Any(s => s.SegmentKind == ODataSegmentKinds.Metadata))
        {
            // Requested metadata
            return GenerateMetadataResponse();
        }

        // not great that these are strings
        if (path.Segments[0].SegmentKind == "entityset")
        {
            MapIEdmEntitySetsToCLR();
            return Buildup(path);
        }
    }
    throw new WebAPIDataServiceException("Can not process OData path", new ODataUnrecognizedPathException());
}

In Buildup that’s where the magic happens. We are going to loop though the path segments to get the keys, navigation and property access then stop once we are past that and let the Web API take over. The path.EdmType is the finial Edm Type we should be returning, which since we only really care if it’s a collection or not we just have to check if it’s a collection. Also I made a weak typed IQueryable FirstOrDefault method.

private IHttpActionResult Buildup(ODataPath path)
{
    var segment = path.Segments[0];
    var edmType = segment.GetEdmType(null);
    var edmCollectionType = edmType as IEdmCollectionType;
    if (edmType == null || edmCollectionType == null ||
        !_edmTypeKindToInterface.ContainsKey(path.EdmType.TypeKind))
    {
        throw new WebAPIDataServiceException("Can not resolve OData Path", new ODataUnrecognizedPathException());
    }
    var query = QueryEntitySet(edmCollectionType);
    for (var i = 1; i < path.Segments.Count; i++)
    {
        edmType = segment.GetEdmType(edmType);
        segment = path.Segments[i];
        // wish this would be an enum.  Don't know if I have all the possibilities set here
        switch (segment.SegmentKind)
        {
            case "key":
                {
                    edmCollectionType = edmType as IEdmCollectionType;
                    var segmentKey = segment as KeyValuePathSegment;
                    query = FilterOnKeys(query, edmCollectionType, segmentKey);
                    break;
                }
            case "navigation":
                {
                    var navigationPath = segment as NavigationPathSegment;
                    query = DrillIntoNavigationProperty(query, navigationPath);
                    break;
                }
            case "property":
                {
                    var propertyPath = segment as PropertyAccessPathSegment;
                    query = ProjectProperty(query, propertyPath);
                    break;
                }
            default:
                {
                    // break out of the loop
                    i = path.Segments.Count + 1;
                    break;
                }
        }
    }
    var contentType = _edmTypeKindToInterface[path.EdmType.TypeKind];
    if (contentType == typeof (IEdmCollectionType))
    {
        // setup to have web api odata take over
        var edmEntityType = GetIEdmTypeToCLRType(GetEdmEntityType(path.EdmType));
        var model = GetMetadata();
        var queryContext = new ODataQueryContext(model, edmEntityType);
        var queryOptions = new ODataQueryOptions(queryContext, Request);
        query = ServiceLocator()
            .Apply(query, queryOptions, new ODataQuerySettings(ODataQuerySettings));
        return new ODataHttpActionResult(this, query, contentType,
                                         typeof (IEnumerable<>).MakeGenericType(query.ElementType));
    }
    return new ODataHttpActionResult(this, FirstOrDefault(query), contentType,
                                     query.ElementType);
}

// have just system type use IEnumerable to create the first one
private static object FirstOrDefault(IQueryable queryable)
{
    var enumerator = queryable.GetEnumerator();
    {
        if (enumerator.MoveNext())
        {
            return enumerator.Current;
        }
    }
    var type = queryable.ElementType;
    return type.IsValueType ? Activator.CreateInstance(type) : null;
}

The FilterOnKeys, DrillIntoNavigationProperty, ProjectProperty is where I build the expression trees and not much to see there, if you are interested you can check out the github repository. I created the IApplyQueryOptions so I could have a default implementation for the Web API but I already have plans to hijack it to allow more control. Also this is the end point of the query before it gets serialized and will allow for changes without have to implement IQueryprovider like we had to do with WCF DataServices

Please feel free to grab the code and test it out. I plan on adding OData Service Operations, the Distinct operator and batching, but that’s for next time.

Monday, May 12th, 2014 OData No Comments

WCF Data Services to remove values from payload with EF provider

One of the more frequent questions about WCF Data Services is how to remove a property from the payload. To actually remove the property from the payload would require implementing IDataServiceMetadataProvider and there is a good blog about doing that here. I started down the road about 9 months ago before ditching it – too much copy and paste from the WCF Data Service core into my implementation and null projections plus a list that kept going on. I’m not going to show how to do that instead I’m going to leave the properties in the payload but not map them. This is probably better since the metadata will not change, but some people think differently.

I want to point out that I tried this about 9 month ago and ditched that as well but this great sample code from Derrick VanArnam showed me the light. Can’t say enough about that code without it I wouldn’t have been able to do this. Some of this code is a direct copy of his sample and some is close but with my tweaks. I’m not trying to pass his great work as mine.

First here is my setup. DataModel context that has a list of customers and their invoices.

public partial class DataModel : DbContext
{
   public DataModel()
       : base("name=DataModel")
   {
   }
    
   protected override void OnModelCreating(DbModelBuilder modelBuilder)
   {
        throw new UnintentionalCodeFirstException();
   }
    
   public IDbSet<Customers> { get; set; }
   public IDbSet<Invoices> { get; set; }
}

public partial class Customer
{
    public Customer()
    {
        this.Invoices = new HashSet();
    }
    
    public string CustomerId { get; set; }
    public string Company { get; set; }
    public string Contact { get; set; }
    public string Title { get; set; }
    public string Address { get; set; }
    public string City { get; set; }
    public string State { get; set; }
    public string Zip { get; set; }
    public string Phone { get; set; }
    public Nullable<CreditLimit> { get; set; }
    public Nullable<Balance> { get; set; }
    
    public virtual ICollection<Invoices> { get; set; }
}

public partial class Invoice
{
    public string InvoiceId { get; set; }
    public Nullable Invoiced { get; set; }
    public string CustomerId { get; set; }
    public string SalesPerson { get; set; }
    public Nullable<Amount> { get; set; }
    public Nullable<Discount> { get; set; }
    public Nullable<Paid> { get; set; }

    public virtual Customer Customers { getset; }
}

Here is my DataService

public class WcfDataService<DataModel> : DataService, IServiceProvider
{
    // This method is called only once to initialize service-wide policies.
    public static void InitializeService(DataServiceConfiguration config)
    {
        config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
        config.SetServiceOperationAccessRule("*", ServiceOperationRights.All);
        config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V3;
        config.UseVerboseErrors = true;
    }

    private readonly DataModel _dataSource;
    private readonly WCFEFProvider _provider;

    public WcfDataService()
    {
        _dataSource = new DataModel();
        _provider = new WCFEFProvider(this, _dataSource);
    }

    public object GetService(Type serviceType)
    {
        if (serviceType.IsInstanceOfType(_provider))
        {
            return _provider;
        }
        return null;
    }

    protected override DataModel CreateDataSource()
    {
        return _dataSource;
    }
}

The WCFEFProvider is my implementation of the AdventureWorksEFProvider in MS samplecode. Also I called my interface IQueryWrapper instead of IObjectQueryWrapper and I just exposed the IQueryable as that’s all that was needed.

public interface IQueryWrapper
{
    IQueryable Query { get; }
}

public class WCFEFProvider<T> : EntityFrameworkDataServiceProvider
{
    /// <param name="service">Provider service</param>
    /// <param name="container">Entity container</param>
    public WCFEFProvider(object service, T container)
        : base(new DataServiceProviderArgs(service, container, null, false))
    {
    }

    /// <summary>
    /// Override the query root
    /// </summary>
    /// <param name="resourceSet"></param>
    /// <returns></returns>
    public override IQueryable GetQueryRootForResourceSet(ResourceSet resourceSet)
    {
        // Parameterize the expression tree
        return WrappedQueryProvider.CreateQuery(base.GetQueryRootForResourceSet(resourceSet));
    }

    /// <summary>
    /// Override the get resource to get underlying ObjectQuery for base.GetResource
    /// </summary>
    /// <param name="query"></param>
    /// <param name="fullTypeName"></param>
    /// <returns></returns>
    public override object GetResource(IQueryable query, string fullTypeName)
    {
        var queryWrapper = query as IQueryWrapper;
        if (queryWrapper != null)
        {
            query = queryWrapper.Query;
        }
        return base.GetResource(query, fullTypeName);
    }

    // We will need to create subclasses for EF to not complain but we need to tell WCF what the ResourceType the subclass is
    public override ResourceType GetResourceType(object target)
    {
        var type = target.GetType();
        var mainclass =
            SecurityProjection.SubClasses.Where(kv => kv.Value == type).Select(kv => kv.Key).FirstOrDefault();
        if (mainclass != null)
        {
            return base.GetResourceType(Activator.CreateInstance(mainclass));
        }
        return base.GetResourceType(target);
    }
}

As a note the GetResource override was what I never figured out 9 months ago and caused me to bail on it. I also rename EFParameterizedQueryProvider to WrappedQueryProvider, EFParameterizedQuery to WrappedIQueryable and made some small tweaks to both.

public class WrappedQueryProvider : IQueryProvider
{
    /// <summary>
    /// Cache the CreateEFParameterizedQuery generic methodinfo
    /// </summary>
    readonly static MethodInfo CreateQueryMethod =
        typeof(WrappedQueryProvider).GetMethod("CreateWrappedIQueryable",
            BindingFlags.Instance | BindingFlags.NonPublic);

    /// <summary>
    /// The underlying Entity Framework query provider
    /// </summary>
    private readonly IQueryProvider _underlyingQueryProvider;

    public WrappedQueryProvider(IQueryProvider underlyingQueryProvider)
    {
        _underlyingQueryProvider = underlyingQueryProvider;
    }

    public static IQueryable CreateQuery(IQueryable underlyingQuery)
    {
       //Wrap it so we can intercept it
        var provider = new WrappedQueryProvider(underlyingQuery.Provider);
        Type elementType = underlyingQuery.Expression.Type.GetQueryElementType();
        return (IQueryable)CreateQueryMethod.MakeGenericMethod(elementType).Invoke(provider, new object[] { underlyingQuery.Expression, underlyingQuery });
    }

    public IQueryable CreateQuery(Expression expression)
    {
        Type elementType = expression.Type.GetQueryElementType();
        return (IQueryable)CreateQueryMethod.MakeGenericMethod(elementType).Invoke(this, new object[] { expression, null });
    }

    public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
    {
        return CreateWrappedIQueryable<TElement>(expression, null);
    }

    private WrappedIQueryable<TElement> CreateWrappedIQueryable<TElement>(Expression expression, IQueryable queryable)
    {
        var objectQuery = queryable as ObjectQuery<TElement> ??
                          (ObjectQuery<TElement>)_underlyingQueryProvider.CreateQuery<TElement>(expression);
        return new WrappedIQueryable<TElement>(objectQuery, this);
    }

    public object Execute(Expression expression)
    {
        //Here we will modify the expressions 
        var securedExpression = new SecurityProjection().CheckSecurity(expression);
        var parameterdExpression = new ParameterizeExpressionVisitor().Parameterize(securedExpression);
        if (typeof(IQueryable).IsAssignableFrom(expression.Type))
        {
            return _underlyingQueryProvider.CreateQuery(parameterdExpression);
        }
        return _underlyingQueryProvider.Execute(parameterdExpression);
    }

    public TResult Execute<TResult>(Expression expression)
    {
        return (TResult)Execute(expression);
    }
}

public class WrappedIQueryable<T> : IOrderedQueryable<T>, IQueryWrapper
{
    /// <summary>
    /// The original Entity Framework ObjectQuery
    /// </summary>
    private readonly IQueryable _queryable;

    /// <summary>
    /// The Entity Framework query provider
    /// </summary>
    private readonly IQueryProvider _queryProvider;

    public WrappedIQueryable(IQueryable<T> objectQuery, IQueryProvider queryProvider)
    {
        _queryProvider = queryProvider;
        _queryable = objectQuery;
    }

    public IEnumerator<T> GetEnumerator()
    {
        return _queryProvider.Execute<IEnumerable<T>>(Expression).GetEnumerator();
    }

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }

    public Type ElementType
    {
        get { return typeof (T); }
    }

    public Expression Expression
    {
        get { return _queryable.Expression; }
    }

    public IQueryProvider Provider
    {
        get { return _queryProvider; }
    }

    IQueryable IQueryWrapper.Query
    {
        get { return _queryable; }
    }
}

You will also need the TypeExtension class from MS sample code (I took it as is).  And since I have all that I took the EFParameterizedExpressionVisitor and called into that to get the benefit of EF parameterizing my OData calls. All the code above code is setup so we can now intercept the Expression Trees.  Now we get into the meat of the issue.

To project I want to do something l like

context.Customers.Select(c=> new Customer()
   {
       CustomerId = c.CustomerId,
       Address = c.Address,
       City = c.City,
       Company = c.Company,
       Contact = c.Contact,
       Phone = c.Phone,
       State = c.State
   });

But Entity Framework will not allow to project to an entity.  The solution I’ve found is it will allow you to project to a subclass

public class CustomersSub : Customers
{
}

// can project
context.Customers.Select(c=> new CustomersSub ()
   {
       CustomerId = c.CustomerId,
       Address = c.Address,
       City = c.City,
       Company = c.Company,
       Contact = c.Contact,
       Phone = c.Phone,
       State = c.State
   });

I also don’t want to create a bunch of empty subclasses by hand or even have T4 template do it.  I’m going to use the TypeBuilder to do it at runtime then cache the result.

public class SecurityProjection : ExpressionVisitor
{
    public static readonly ConcurrentDictionary<Type, Type> SubClasses = new ConcurrentDictionary<Type, Type>();
    private static readonly ModuleBuilder _moduleBuilder;

    // These will need to be changed based on your requirements of when properties are removed or not
    protected static readonly IDictionary<Type, IList<string>> removeProperties = new Dictionary<Type, IList<string>>();

    static SecurityProjection()
    {
        // Will need these to create subclasses on the fly
        var assemblyName = new AssemblyName("SecurityProjectionAssembly");
        var assemblyBuilder = AppDomain.CurrentDomain.DefineDynamicAssembly(assemblyName, AssemblyBuilderAccess.Run);
        _moduleBuilder = assemblyBuilder.DefineDynamicModule("SecurityProjectionModule");

        // always remove the credit limit and balance & sales person
        removeProperties.Add(typeof(Customer), new List<string>());
        removeProperties[typeof(Customer)].Add("CreditLimit");
        removeProperties[typeof(Customer)].Add("Balance");
        removeProperties.Add(typeof(Invoice), new List<string>());
        removeProperties[typeof(Invoice)].Add("SalesPerson");

    }

    public Expression CheckSecurity(Expression expression)
    {
        expression = Visit(expression);
        return expression;
    }

    protected override MemberBinding VisitMemberBinding(MemberBinding node)
    {
        var memberAssignment = node as MemberAssignment;
        if (memberAssignment != null)
        {
            var memType = memberAssignment.Expression.Type;
            if (memType != null)
            {
                if (removeProperties.ContainsKey(memType))
                {
                    // make a subclass for the projection as EF will not allow you project to entity
                    var to = SubClasses.GetOrAdd(memType, CreateSubClass);

                    var projection = Project(memType, to, memberAssignment.Expression);
                    var binder = Expression.Bind(memberAssignment.Member, projection);
                    return binder;
                }
                memType = memType.GetQueryElementType();
                if (memType !- null && removeProperties.ContainsKey(memType))
                {
                    // make a subclass for the projection as EF will not allow you project to entity
                    var to = SubClasses.GetOrAdd(memType, CreateSubClass);

                    // parameter of the expression
                    var source = Expression.Parameter(memType, "source");
                    var projection = Project(memType, to, source);

                    var func = Expression.Lambda(typeof(Func<,>).MakeGenericType(memType, to), projection, source);

                    var result = Expression.Call(typeof(Enumerable), "Select", new[] { memType, to }, memberAssignment.Expression, func);
                    var binder = Expression.Bind(memberAssignment.Member, result);
                    return binder;
                }
            }
        }

        return base.VisitMemberBinding(node);
    }

    protected override Expression VisitMethodCall(MethodCallExpression node)
    {
        var methodType = CheckIQueryable(node.Method.ReturnType);
        if (methodType != null && removeProperties.ContainsKey(methodType))
        {
            // make a subclass for the projection as EF will not allow you project to entity
            var to = SubClasses.GetOrAdd(methodType, CreateSubClass);
            // parameter of the expression
            var source = Expression.Parameter(methodType, "source");
            var projection = Project(methodType, to, source);

            var func = Expression.Lambda(typeof (Func<,>).MakeGenericType(methodType, to), projection, source);

            var result = Expression.Call(typeof (Queryable), "Select", new[] {methodType, to}, node, func);
            return result;
        }

        return base.VisitMethodCall(node);
    }

    private Type CheckIQueryable(Type type)
    {
        return
            type.GetInterfaces()
                .Where(i => i.IsGenericType && i.GetGenericTypeDefinition() == typeof (IQueryable<>))
                .Select(i => i.GetGenericArguments().First())
                .FirstOrDefault();
    }

    private Type CreateSubClass(Type type)
    {
        var subclass = _moduleBuilder.DefineType(type.Name + "SubClass", type.Attributes, type);
        var newType = subclass.CreateType();
        return newType;
    }

    private Expression Project(Type from, Type to, Expression pSource)
    {
        var ignoreProps = removeProperties[from];

        var bindings = GetProperties(from, false)
                    .Join(GetProperties(to, true), source => new {source.Name, source.PropertyType},
                            dest => new {dest.Name, dest.PropertyType}, (source, dest) => new {source, dest})
                    .Where(a => !ignoreProps.Contains(a.source.Name))
                    .Select(prop => Expression.Bind(prop.dest, Expression.Property(pSource, prop.source)))
                    .ToList();
        return Expression.MemberInit(Expression.New(to), bindings);
    }

    private IEnumerable<PropertyInfo> GetProperties(Type type, bool write)
    {
        var properties = type.GetProperties(BindingFlags.Public | BindingFlags.Instance).AsEnumerable();
        if (write)
        {
            properties = properties.Where(p => p.CanWrite);
        }
        else
        {
            properties = properties.Where(p => p.CanRead);
        }

        return properties;
    }
}

Something’s to explain. I created a static IDictionary> to hold the property names I wanted to remove from the projection for that type. I have it hard coded to always remove the CreditLimit and Balance from the Customer and from the invoice to not show the sales person. The visit method call we are looking for the root query to do the projection similar above to how I wrote it out by hand. Then VisitMemberBinding to look for any types that are either the type we are looking for or IEnumberable of the type we are looking for. If it’s just the type we create a new subclass type in it’s place. For IEnumberables we do a select to project it to our subclass. This is why we needed to create an empty entity to pass down in the WCFEFProvider GetResourceType method. Otherwise WCF will be looking for an entity of the type of the subclass and not the original entity.

It would probably be worth caching the property of each type instead of using reflection each time.

Now when hitting the webservice Invoices will still return the sales person but it will be null and the same for balance and credit limit for the customer. I’ve tested this with projection, expand and it all seems to be working but I can’t say it’s 100% bullet proof. 🙂

Tags: , ,

Friday, April 25th, 2014 OData Comments Off on WCF Data Services to remove values from payload with EF provider

Processing tasks as they complete with Rx

Microsoft has a couple of articles on processing tasks as they complete, instead of waiting for Task.WhenAll: Start Multiple Async Tasks and Process Them As They Complete & Processing tasks as they complete.  I’m going to show how to do it with Reactive Extensions.  A way to think about IObservable is it’s a future IEnumerable and what is a Task[]?  It’s a future IEnumerable!

For this example I’m going to use a Console application and hit an OData endpoint.

First we are going to create a class that will query the OData endpoint figure out how many calls we need to make then return back an array of Tasks.  The method signature looks a little funky because OData Task returns back an IEnumberable<T>.

public class CombineOData
{
    private readonly DataServiceContext _dataServiceContext;
    private const double PageSize = 250L;

    public CombineOData(Uri oDataSite)
    {
        _dataServiceContext = new DataServiceContext(oDataSite, DataServiceProtocolVersion.V3);
    }

    public async Task<Task<IEnumerable>[]> GetData(string entityName)
    {
        var query = _dataServiceContext.CreateQuery(entityName);
        // Get the total number of entities in the feed
        query = query.AddQueryOption("$inlinecount", "allpages")
                     .AddQueryOption("$top", PageSize);
            
        // need to make the first call to just get the total count
        var webCall = await Task.Factory.FromAsync<IEnumerable>(query.BeginExecute, query.EndExecute, null);
        var totalCount = ((QueryOperationResponse) webCall).TotalCount;

        // Store the result into a new task so we don't hit the website again for data we already have
        var result = new [] {Task.FromResult(webCall)};

        // If there are more entities than asked for create them all
        if (totalCount > PageSize)
        {
            result = AllWebCalls(totalCount, entityName).Concat(result).ToArray();
        }

        return result;
    }

    private IEnumerable<Task<IEnumerable>> AllWebCalls(long entityCount, string entityName)
    {
        // get the number of entities we need to download per chunk
        var numberOfCalls = Convert.ToInt32(Math.Ceiling(entityCount / PageSize) - 1);
        for (var i = 0; i < numberOfCalls; i++)
        {
            var query =
                _dataServiceContext.CreateQuery(entityName)
                                   .AddQueryOption("$top", PageSize)
                                   .AddQueryOption("$inlinecount", "none")
                                   .AddQueryOption("$skip", PageSize*(i + 1));

            yield return
                Task.Factory.FromAsync<IEnumerable>(query.BeginExecute, query.EndExecute, null);

        }
    }
}

We also need a class to deserialize the OData call into.

public class ProductCatalog
{
    public DataServiceStreamLink ThumbNailPhoto { get; set; }
    public DataServiceStreamLink LargePhoto { get; set; }
    public int ProductID { get; set; }
    public string ProductNumber { get; set; }
    public string ProductName { get; set; }
    public string ProductModel { get; set; }
    public string ProductCategory { get; set; }
    public string ProductSubcategory { get; set; }
    public string Description { get; set; }
    public string CultureID { get; set; }
    public string Color { get; set; }
    public string Size { get; set; }
    public decimal? Weight { get; set; }
    public decimal ListPrice { get; set; }
    public long ID { get; set; }
}

Now in our console application we will convert the task into IObservables and process them when they come in.

private static void Main(string[] args)
{
    var task = Task.Run(async () =>
        {
            var uri =
                new Uri("http://services.odata.org/AdventureWorksV3/AdventureWorks.svc/");
            var client = new CombineOData(uri);

            // products is an array of task that return IEnumerables.
            var productsTasks = await client.GetData("ProductCatalog");

            // convert the task to observable 
            // then merge them into one observable
            // then make the IEnumberable into an observable
            var products = productsTasks.Select(p => p.ToObservable())
                                        .Merge()
                                        .SelectMany(t => t.ToObservable());

            // want to IConnectable because we will subscribe twice one for the result and await for the results to be done
            var publisher = products.Publish();
                    
            // just log to the console window to view the results as they come in to prove it's processing it as they arrive
            publisher.Subscribe(
                      catalog => Console.WriteLine("{0} {1} {2} {3}", catalog.ID, catalog.ProductID, catalog.ProductName,
                                                   catalog.ProductNumber), 
                      ex => Console.WriteLine(ex.Message), 
                      () => Console.WriteLine("Completed"));

            // Don't continue past this point until the observable is complete
            // this will also auto connect the publisher and start the process
            var data = await publisher;

            // proof that we didn't hit this line  until afterwards
            Console.WriteLine("After await {0}", data.ID);
        });
            
    Console.ReadKey();
}

The real key is to switch the task ToObservable and the merge to make them one sequence. From that we just needed to publish so we can have our method await for it to finish.

Tags: , , ,

Tuesday, April 22nd, 2014 Reactive Extensions No Comments

Move from WCF Data Services to Web API – Part 4

In Part 2 in the method QueryEntitySet I had a subtle bug.

                var process = (Func<IEdmEntityType, IQueryable>)
                              _typeToGeneric.GetOrAdd(new KeyValuePair<IEdmEntityType, Type>(edmType, entityType),
                                                      pair =>
                                                      {
                                                          var method = ((MethodInfo)
                                                                        MethodBase.GetMethodFromHandle(
                                                                            _processRequest, GetType().TypeHandle))
                                                              .MakeGenericMethod(pair.Value);
                                                          return
                                                              Delegate.CreateDelegate(
                                                                  typeof(Func<IEdmEntityType, IQueryable>), this,
                                                                  method);
                                                      });

In the Delegate.CreateDelegate I’m passing in “this”. That will create a closures around the first controller and will call into that controller object every time. This is not what we want because that controller would be disposed and plus we now have an object that’s just hanging around. I’m going to fix that with ExpressionTrees by passing in the current controller object.

        private Func, IEdmEntityType, IQueryable> SwitchToGenericMethod(
            IEdmEntityType edmEntityType, Type clrType)
        {
            return
                (Func, IEdmEntityType, IQueryable>)
                _typeToGeneric.GetOrAdd(new KeyValuePair(edmEntityType, clrType),
                                        pair =>
                                            {
                                                var method = ((MethodInfo)
                                                              MethodBase.GetMethodFromHandle(
                                                                  _processRequest, GetType().TypeHandle))
                                                    .MakeGenericMethod(pair.Value);
                                                var controller =
                                                    Expression.Parameter(
                                                        typeof (ODataServicesController),
                                                        "oDServiceController");
                                                var edmentityType =
                                                    Expression.Parameter(typeof (IEdmEntityType),
                                                                         "edmEntityType");

                                                var expr = Expression.Call(controller, method, edmentityType);
                                                return
                                                    Expression.Lambda,
                                                        IEdmEntityType, IQueryable>>(expr, controller,
                                                                                     edmentityType).Compile();
                                            });
        }

Now in QueryEntitySet we change the code to be the following

                // Switch from system Type to generic type
                var process = SwitchToGenericMethod(edmType, entityType);
                var query = process(this, edmType);

Also we need to map all the entities to the CLR type otherwise $expand and $link won’t work.

        private void MapIEdmEntitySetsToCLR()
        {
            var metaData = GetMetadata();
            var entitySets = Container.EntitySets();
            foreach (var entitySet in entitySets)
            {
                var entityType = GetIEdmTypeToCLRType(entitySet.ElementType);
                metaData.SetAnnotationValue(entitySet.ElementType, new ClrTypeAnnotation(entityType));
            }
        }

        private IEdmEntityContainer Container
        {
            get
            {
                var metadata = GetMetadata();
                return metadata.EntityContainers().First();
            }
        }

I removed in the ProcessRequest method this code “model.SetAnnotationValue(edmEntityType, new ClrTypeAnnotation(typeof (TEntity)));” and replaced it with a call into MapIEdmEntitySetsToCLR().

From my testing how I’m determining if we are returning an EntitySet or Entity isn’t correct in the ProcessRequest method. If I ask for something like \api\odata\Customer(100)\SalesOrders the request will be set as a EntitySet and just return all the SalesOrders. Not filtered down to the one customer. So I will need to create, or find if Web API, has something to help me out. I’m doubtful about Web API because they are using controllers and actions to handle this and we want to just generate the ExpressionTrees automatically. I will be looking at the source code for WCF Data Services to see what I can “borrow” from them. I don’t think they will be too upset since they are getting ready to open source it. Because of this it might be a couple of days before I get to Part 5. But I wanted to update everyone along the way of things I found I needed to change. That’s part of the fun of writing the blog while I’m still working out the kinks and not at the end where everything is working 🙂

Tags: , ,

Friday, April 18th, 2014 OData No Comments

Move from WCF Data Services to Web API – Part 3

See Part 1 and Part 2.  I know in Part 2 I said I was going to look at ExpressionTrees and limit down to a single entity but I wanted to fix the Content Negotiation and add the service document.  Both turned out to be simple, once I discovered the Media Type you pass into the ODataMediaTypeFormatter doesn’t do anything.

First I created another method to return back the correct MediaTypeFormatter

        private MediaTypeFormatter GetFormatter(Type objType)
        {
            var conneg = Configuration.Services.GetContentNegotiator();
            var result = conneg.Negotiate(objType, Request, Configuration.Formatters);
            return result.Formatter;
        }

Then changed GenerateMetadataResponse to this

        protected HttpResponseMessage GenerateMetadataResponse()
        {
            var odataMediaTypeFormatter = GetFormatter(typeof (IEdmModel));
            var formatter = odataMediaTypeFormatter.GetPerRequestFormatterInstance(typeof (IEdmModel), Request,
                                                                                   null);
            var response = Request.CreateResponse();
            response.Content = new ObjectContent(typeof(IEdmModel), GetMetadata(), formatter);
            return response;
        }

Now it’s not hard coded to XML but still returns XML.  What’s still missing if you just say you want JSON it doesn’t error like WCF Data Services did it will just return XML.  I think it should error (add to the list)

This is the bottom part of QueryEntitySet

                var response = Request.CreateResponse();
                var resultType = typeof(IEnumerable<>).MakeGenericType(query.ElementType);
                var odataMediaTypeFormatter = GetFormatter(resultType);
                var formatter = odataMediaTypeFormatter.GetPerRequestFormatterInstance(typeof(IEdmCollectionType), Request, null);
                response.Content = new ObjectContent(resultType, query, formatter);

                return response;

Pretty straight forward.   I also created the Service Document for the OData.  That turned out to be simple.

        private HttpResponseMessage GenerateServiceDocument()
        {
            var response = Request.CreateResponse();
            var odataMediaTypeFormatter = GetFormatter(typeof(ODataWorkspace));
            var formatter = odataMediaTypeFormatter.GetPerRequestFormatterInstance(typeof(ODataWorkspace), Request, null);
            response.Content = new ObjectContent(typeof(ODataWorkspace), GetServiceDocument(), formatter);
            return response;
        }

This is ProcessRequest now

        protected virtual HttpResponseMessage ProcessRequest()
        {
            var pathHandler = Request.GetODataPathHandler();
            var metadata = GetMetadata();
            var path = pathHandler.Parse(metadata, GetODataPath());
            Request.SetODataPath(path);
            Request.SetODataRouteName("OData");
            if (path.Segments.Count == 0)
            {
                // Requested service document
                return GenerateServiceDocument();
            }

            if (path.Segments.Any(s => s.SegmentKind == ODataSegmentKinds.Metadata))
            {
                // Requested metadata
                return GenerateMetadataResponse();
            }

            var collectionType = path.EdmType as IEdmCollectionType;
            if (collectionType != null)
            {
                // Requested entity collection
                return QueryEntitySet(collectionType);
            }

            return new HttpResponseMessage(HttpStatusCode.NotImplemented);
        }

Now in Part 4 I’ll get to working on the ExpressionTrees.

Tags: , ,

Wednesday, April 16th, 2014 OData Comments Off on Move from WCF Data Services to Web API – Part 3

Move from WCF Data Services to Web API – Part 2

From Part 1 of getting Metadata up and running we are now going to work on getting querying an entity set working.  But first a quick update.  From the last time I left getting the metadata in the processing request and I did move that out into it’s own method.

        protected HttpResponseMessage GenerateMetadataResponse()
        {
            var odataMediaTypeFormatter = new ODataMediaTypeFormatter(new DefaultODataDeserializerProvider(),
                                                                          new DefaultODataSerializerProvider(),
                                                                          new[] { ODataPayloadKind.MetadataDocument });

            //ToDo will need to make sure they can accept application/xml
            var formatter = odataMediaTypeFormatter.GetPerRequestFormatterInstance(typeof(IEdmModel), Request,
                                                                                   new MediaTypeHeaderValue(
                                                                                       "application/xml"));
            var response = Request.CreateResponse();
            response.Content = new ObjectContent(typeof(IEdmModel), GetMetadata(), formatter);
        }

Now the first thing we need to do is see if we are dealing with an entity set.

            var collectionType = path.EdmType as IEdmCollectionType;
            if (collectionType != null)
            {
                return QueryEntitySet(collectionType);
            }

If collectionType isn’t null then we will need to check for and retrieve the EntityType.

        private IEdmEntityType GetEdmEntityType(IEdmType edmType)
        {
            var edmEntityType = edmType as IEdmEntityType;
            if (edmEntityType == null)
            {
                var collectionType = edmType as IEdmCollectionType;
                if (collectionType != null)
                {
                    edmEntityType = collectionType.ElementType.AsEntity().EntityDefinition();
                }
            }

            return edmEntityType;
        }

If this method returns null then we are not dealing with an entity or entity set.  From there we will need to switch from IEdmEntityType to the CLR type.   For that I’m going to create another interface and create a default implementation for DbContext.

namespace ODataServices.Interfaces
{
    public interface IEdmEntityToClrConverter
    {
        Type AsClrType<TSource>(TSource source, IEdmEntityType edmEntityType);
    }
}

and here is the default implementation.  I’m not sold on how I’m looking up the CLR type from metadata of the DbContext and if someone else has a better idea let me know.  Or even better answer this stackoverflow question.

namespace ODataServices
{
    public class DbContextEdmEntityToClrConverter : IEdmEntityToClrConverter
    {
        private readonly static ConcurrentDictionary<IEdmEntityType, Type> _cachedConversions =
            new ConcurrentDictionary<IEdmEntityType, Type>();

        public Type AsClrType<TSource>(TSource source, IEdmEntityType edmEntityType)
        {
            var dbContext = source as DbContext;
            if (dbContext == null)
            {
                return null;
            }

            // Can't use GetOrAdd want to trap for null
            Type result;
            if (!_cachedConversions.TryGetValue(edmEntityType, out result))
            {
                result = ConvertIEdmEntityTypeToClr(edmEntityType, dbContext);
                if (result != null)
                {
                    _cachedConversions.TryAdd(edmEntityType, result);
                }
            }

            return result;
        }

        private Type ConvertIEdmEntityTypeToClr(IEdmEntityType edmEntityType, DbContext context)
        {
            var metadata = ((IObjectContextAdapter)context).ObjectContext.MetadataWorkspace;
            var oSpace = metadata.GetItemCollection(DataSpace.OSpace);
            var typeName = oSpace.GetItems<EntityType>().Select(e => e.FullName).FirstOrDefault(name =>
            {
                var fullname = name + ":" + edmEntityType.FullName();
                MappingBase map;
                return metadata.TryGetItem(fullname, DataSpace.OCSpace, out map);
            });

            if (typeName != null)
            {
                return Type.GetType(typeName, null, GetTypeFromAssembly, false, false);
            }
            return null;
        }

        private Type GetTypeFromAssembly(Assembly assembly, string nameOfType, bool ignoreCase)
        {
            if (assembly == null)
            {
                var resolver = new DefaultAssembliesResolver();
                return resolver.GetAssemblies()
                               .Select(a => a.GetType(nameOfType, false, ignoreCase))
                               .FirstOrDefault(t => t != null);
            }
            return assembly.GetType(nameOfType, false, ignoreCase);
        }
    }
}

Noticed I had to  use an Assembly Resolver otherwise Type.GetType() would fail.  Now that we have a type I’m going to use a bit of reflection and switch us from a System Type to a generic type.  I’m going to store a method handle to the method we want to switch to and grab that in the static constructor since it shouldn’t change. Also I’m going to add a couple more interfaces we haven’t talk about yet but will quickly discuss.

        static ODataServicesController()
        {
            _processRequest =
                typeof (ODataServicesController<TSource>).GetMethods(BindingFlags.Instance | BindingFlags.NonPublic)
                                                         .First(
                                                             m =>
                                                             m.Name == "ProcessRequest" && m.IsGenericMethodDefinition)
                                                         .MethodHandle;

            // Load up default implementations
            _serviceLocatorDefaults[typeof (IEdmModelFactory)] =
                new Lazy<object>(() => new DbContextEdmModel());
            _serviceLocatorDefaults[typeof (IEdmEntityToClrConverter)] =
                new Lazy<object>(() => new DbContextEdmEntityToClrConverter());
            _serviceLocatorDefaults[typeof (IQueryRootProvider)] =
                new Lazy<object>(() => new DbContextQueryRoot());
        }

        private static readonly RuntimeMethodHandle _processRequest;

IQueryRootProvider will be used to return the starting point of the IQueryable.  IQueryInterceptor is used to add a where clause.  I was on the fence on that interface as it could be implemented in IQueryRootProvider but added it for flexibility and since WCF Data Services has something similar.

namespace ODataServices.Interfaces
{
    public interface IQueryRootProvider
    {
        IQueryable<TEntity> QueryRoot<TSource, TEntity>(TSource source)
            where TEntity : class;
    }
}

namespace ODataServices.Interfaces
{
    public interface IQueryInterceptor
    {
        Expression<Func<TEntity, bool>> Intercept<TEntity>();
    }
}

Here is the default implementation of IQueryRootProvider for DbContext

namespace ODataServices
{
    public class DbContextQueryRoot : IQueryRootProvider
    {
        public IQueryable<TEntity> QueryRoot<TSource, TEntity>(TSource source)
            where TEntity : class
        {
            var dbContext = source as DbContext;
            if (dbContext == null)
            {
                return null;
            }

            return QueryRoot<TEntity>(dbContext);
        }

        private IQueryable<TEntity> QueryRoot<TEntity>(DbContext dbContext)
            where TEntity : class
        {
            return dbContext.Set<TEntity>().AsNoTracking();
        }
    }

Pretty simple stuff. For IQueryInterceptor I created a default implementation but it doesn’t rely on DbContext. First I created an attribute called QueryInterceptorAttribute, same name as WCF Data Services.

namespace ODataServices.Attributes
{
    [AttributeUsageAttribute(AttributeTargets.Method, AllowMultiple = true, Inherited = true)]
    public class QueryInterceptorAttribute : Attribute
    {
    }
}

namespace ODataServices
{
    public abstract class QueryInterceptor : IQueryInterceptor
    {
        private readonly MethodInfo[] _methodInfos;

        protected QueryInterceptor()
        {
            _methodInfos = GetType().GetMethods()
                                    .Where(
                                        m =>
                                        m.GetCustomAttributesData()
                                         .Any(a => a.AttributeType == typeof(QueryInterceptorAttribute)) &&
                                        !m.GetParameters().Any() &&
                                        m.IsGenericMethod == false).ToArray();
        }

        public Expression<Func<TEntity, bool>> Intercept<TEntity>()
        {
            var inteceptor = _methodInfos.FirstOrDefault(m => m.ReturnType == typeof(Expression<Func<TEntity, bool>>));
            if (inteceptor != null)
            {
                return inteceptor.Invoke(this, new object[0]) as Expression<Func<TEntity, bool>>;
            }
            return null;
        }
    }
}

With the abstract class of QueryInterceptor you can create a class that inherits from it and mark methods with the QueryInterceptor attribute. If the return type of the method is type we are looking for it will auto add the return value to the where clause.  If you don’t like this implementation you can just create a different class that implements IQueryInterceptor, that’s the great part.  Also word of warning I haven’t tested that code yet 🙂

Now to put it all together

        protected HttpResponseMessage QueryEntitySet(IEdmCollectionType edmCollectionType)
        {
            var edmType = GetEdmEntityType(edmCollectionType);
            if (edmType != null)
            {
                var entityType = GetIEdmTypeToCLRType(edmType);
                // Switch from system Type to generic type
                var process = (Func<IEdmEntityType, IQueryable>)
                              _typeToGeneric.GetOrAdd(new KeyValuePair<IEdmEntityType, Type>(edmType, entityType),
                                                      pair =>
                                                      {
                                                          var method = ((MethodInfo)
                                                                        MethodBase.GetMethodFromHandle(
                                                                            _processRequest, GetType().TypeHandle))
                                                              .MakeGenericMethod(pair.Value);
                                                          return
                                                              Delegate.CreateDelegate(
                                                                  typeof(Func<IEdmEntityType, IQueryable>), this,
                                                                  method);
                                                      });
                var query = process(edmType);
                var response = Request.CreateResponse();
                var odataMediaTypeFormatter = new ODataMediaTypeFormatter(new DefaultODataDeserializerProvider(),
                                                                          new DefaultODataSerializerProvider(),
                                                                          new[] { ODataPayloadKind.Feed });

                //ToDo will need to use a content negotiator - hard code to XML for now
                var formatter = odataMediaTypeFormatter.GetPerRequestFormatterInstance(typeof(IEdmCollectionType), Request,
                                                                                       new MediaTypeHeaderValue(
                                                                                           "application/xml"));
                response.Content = new ObjectContent(typeof(IEnumerable<>).MakeGenericType(query.ElementType), query, formatter);
                return response;
            }
            throw new Exception("Entity Set not found!");
        }

        private IQueryable ProcessRequest<TEntity>(IEdmEntityType edmEntityType)
            where TEntity : class
        {

            var queryable = GetQueryRoot<TEntity>();

            // Check if there are any interceptor
            var interception = GetInterceptor<TEntity>();
            if (interception != null)
            {
                queryable = queryable.Where(interception);
            }

            var model = GetMetadata();

            // need to tell Web API about the mapping between IEdmEntityType and the CLR Type
            model.SetAnnotationValue(edmEntityType, new ClrTypeAnnotation(typeof (TEntity)));
            var queryContext = new ODataQueryContext(model, typeof (TEntity));
            var queryOptions = new ODataQueryOptions(queryContext, Request);

            //ToDo this does the default Web API filtering but would like to get more control over it
            return queryOptions.ApplyTo(queryable);
        }

        private IQueryable<TEntity> GetQueryRoot<TEntity>()
            where TEntity : class
        {
            var queryRoot = ServiceLocator<IQueryRootProvider>();
            if (queryRoot != null)
            {
                return queryRoot.QueryRoot<TSource, TEntity>(CurrentDataSource);
            }
            return null;
        }

        private Expression<Func<TEntity, bool>> GetInterceptor<TEntity>()
        {
            var interception = ServiceLocator<IQueryInterceptor>();
            if (interception != null)
            {
                return interception.Intercept<TEntity>();
            }
            return null;
        }

This is what the ProcessRequest method looks like now and I added to dispose of the Data Source.

        protected virtual HttpResponseMessage ProcessRequest()
        {
            var pathHandler = Request.GetODataPathHandler();
            var metadata = GetMetadata();
            var path = pathHandler.Parse(metadata, GetODataPath());
            Request.SetODataPath(path);
            Request.SetODataRouteName("OData");
            if (path.Segments.Any(s => s.SegmentKind == ODataSegmentKinds.Metadata))
            {
                return GenerateMetadataResponse();
            }

            var collectionType = path.EdmType as IEdmCollectionType;
            if (collectionType != null)
            {
                return QueryEntitySet(collectionType);
            }

            return new HttpResponseMessage(HttpStatusCode.NotImplemented);
        }

        protected override void Dispose(bool disposing)
        {
            if (disposing)
            {
                if (_currentDataSource.IsValueCreated)
                {
                    var disposer = CurrentDataSource as IDisposable;
                    if (disposer != null)
                    {
                        disposer.Dispose();
                    }
                }
            }
            base.Dispose(disposing);
        }

Now we should be able to hit /api/odata/Customers?$orderby=City&$select=City or /api/odata/Customers. Assuming your DbContext has an Customer Entity Set and property called city 🙂 Next I will be trying to get a single entity but that’s for a Part 3 and we’ll need to dive into ExpressionTrees.

Tags: , ,

Tuesday, April 15th, 2014 OData Comments Off on Move from WCF Data Services to Web API – Part 2

Move from WCF Data Services to Web API

If you haven’t heard WCF Data Services is going to be put into maintenance mode.  🙁 While I have always had a love/hate relationship with it, it provided a convenient way to expose data as OData.   The new way is to use Web API OData support but that doesn’t quite work for our needs.   For one we would have to write a ton of boiler plate code and keep all that code in sync with our data model.   From the comment section it was discussed at writing a handler to fill this gap.  This is my attempt at doing that.  By no means is this done and it’s still a work in progress but I figured I would take any reader along for the journey.  Maybe I can get some better ways to solve a problem or better alternatives from readers since this is so new.

Some of my goals

  • Not having the handler tied to Entity Framework.
  • Be extensible, something the WCF Data Services was lacking.
  • Use Web API OData support as much as possible
  • For phase 1 just looking at getting query support.
  • Not looking at batch support for phase 1.

Enough of that lets get to code.   First I created Web API solution called it WebAPIOData.  The standard project of WebAPIOData will be where I’m testing/exploring the options from my handler.  To this solution I added another projected call ODataServices and made that an empty web project.  Then removed the global.asax and web.configs.

First off I couldn’t use the Web API OData routing because I don’t have my IEdmModel at startup time and can’t create my DbContext at startup time.  Our application uses Entity Framework code first with MEF to dynamically build the model at runtime, along with user security.  That ruled out a static model at startup time, WCF Data Services allowed us to create the data source at runtime.

In the App_Stsart\WebAPiConfig class I added a new HttpRoute.

 

            config.Routes.MapHttpRoute("OData", "api/odata/{*wildcard}",
                                       new
                                           {
                                               controller = "ODService",
                                               action = "Get",
                                               wildcard = RouteParameter.Optional
                                           },
                                       new {httpMethod = new HttpMethodConstraint(new HttpMethod("GET"))});
            config.Routes.IgnoreRoute("ODataIgnore", "api/ODService/{*wildcard}");

I’m going to create a controller called ODServiceController that I want to map to api/odata.  I also want to stop anyone from going to it directly which is why I configured Web API to ignore it.   One of the most important parts is the {*wildcard} since I want all calls to be directed to it and not have MVC trying to parse it out for an action method.  Right now it’s restricted to Get and will always call the Get method.

Now in the ODataServices project I created a folder called Controllers and updated to Web API 2 and added references to Entity Framework 6.1 then created an abstract generic controller that inherits from ODataMetadataController called ODataServicesController

One of the first things I’m going to do is setup a Service Locator, I know a lot of people consider this an anti-pattern but when building tools it’s hard to use a real IOC container.

 

namespace ODataServices.Controllers
{
    public abstract class ODataServicesController<TSource> : ODataMetadataController
    {
        protected ODataServicesController()
        {
            SetServiceLocator();
        }

        #region ServiceLocator
        private static IDictionary<Type, Lazy<object>> _serviceLocatorDefaults =
            new Dictionary<Type, Lazy<object>>();

        private Func<Type, object> _serviceLocator;

        private T ServiceLocator<T>() where T : class
        {
            return _serviceLocator(typeof (T)) as T;
        }

        private void SetServiceLocator()
        {
            // check if datasource or controller implements IServiceProvider;
            var dataSourceResolver = (typeof (IServiceProvider).IsAssignableFrom(typeof (TSource)));
            var controllerResolver = (typeof (IServiceProvider).IsAssignableFrom(GetType()));

            if (dataSourceResolver && controllerResolver)
            {
                _serviceLocator = type =>
                    {
                        var result = ((IServiceProvider) this).GetService(type);
                        if (result == null)
                        {
                            result = ((IServiceProvider) CurrentDataSource).GetService(type);
                            if (result == null)
                            {
                                result = DependencyResolver.Current.GetService(type);
                            }
                            if (result == null && _serviceLocatorDefaults.ContainsKey(type))
                            {
                                result = _serviceLocatorDefaults[type].Value;
                            }
                        }
                        return result;
                    };
            }
            else if (dataSourceResolver)
            {
                _serviceLocator = type =>
                    {
                        var result = ((IServiceProvider) CurrentDataSource).GetService(type);
                        if (result == null)
                        {
                            result = DependencyResolver.Current.GetService(type);
                        }
                        if (result == null && _serviceLocatorDefaults.ContainsKey(type))
                        {
                            result = _serviceLocatorDefaults[type].Value;
                        }
                        return result;
                    };
            }
            else if (controllerResolver)
            {
                _serviceLocator = type =>
                    {
                        var result = ((IServiceProvider) this).GetService(type);
                        if (result == null)
                        {
                            result = DependencyResolver.Current.GetService(type);
                        }
                        if (result == null && _serviceLocatorDefaults.ContainsKey(type))
                        {
                            result = _serviceLocatorDefaults[type].Value;
                        }
                        return result;
                    };
            }
            else
            {
                _serviceLocator = type =>
                    {
                        var result = DependencyResolver.Current.GetService(type);
                        if (result == null && _serviceLocatorDefaults.ContainsKey(type))
                        {
                            result = _serviceLocatorDefaults[type].Value;
                        }
                        return result;
                    };
            }
        }
        #endregion
    }
}

From the code above I’m checking if the controller implements IServiceProvider then check if it can provide the object.  If the controller either doesn’t implement the interface or returns null then to check the datasource.  If the data source doesn’t implement IServiceProvider or returns null then check the standard Web API Dependency Resolver.  If all that fails then we have a dictionary that will contain a mapping of defaults that we will define for the handler.   This should give us the extensibility that we need.

Now we are going to create a method for creating the data source.  Here I’m going to use the same methodology that WCF Data Services did.  We will create a method called CreateDataSource and a property called CurrentDataSource.

 

        protected ODataServicesController()
        {

            SetServiceLocator();
            // Setup data source to get resolved first time needed.
            _currentDataSource = new Lazy<TSource>(CreateDataSource, LazyThreadSafetyMode.ExecutionAndPublication);
        }

        #region DataSource
        private readonly Lazy<TSource> _currentDataSource;

        protected TSource CurrentDataSource
        {
            get { return _currentDataSource.Value; }
        }

        protected virtual TSource CreateDataSource()
        {
            return DependencyResolver.Current.GetService<TSource>();
        }
        #endregion

Here I’m taking advantage of the .Net Lazy class and using that to automatically call CreateDataSource() first time we need it.  Then I just write a wrapper property on  CurrentDataSource to retrieve the value property of the Lazy backing field.  Pretty simple but should flow nicely.

Now on to the meat of the controller. In the ODataServicesController I’m going to create a method called ProcessRequest that returns an HttpResponseMessage.   The first thing we need to do is get the ODataPath which the Request property has an extension method just for that.  Then we need to get our IEdmModel and parse it out.

        protected override void Initialize(HttpControllerContext controllerContext)
        {
            base.Initialize(controllerContext);
            Request.SetEdmModel(BuildEdmModel());
        }

        protected virtual HttpResponseMessage ProcessRequest()
        {
            var pathHandler = Request.GetODataPathHandler();
            var metadata = GetMetadata();
            var path = pathHandler.Parse(metadata, GetODataPath());
            Request.SetODataPath(path);

            return null;
        }

        protected virtual string GetODataPath()
        {
            var routedata = Request.GetRouteData();
            var uriTemplate = new UriTemplate(routedata.Route.RouteTemplate);
            var baseUri = new Uri(Request.RequestUri.Scheme + "://" +
                                  Request.RequestUri.Authority +
                                  Request.GetRequestContext().VirtualPathRoot.TrimEnd('/') + "/");
            var match = uriTemplate.Match(baseUri, Request.RequestUri);
            var path = "/" + String.Join("/", match.WildcardPathSegments);
            return path;
        }

        private IEdmModel BuildEdmModel()
        {

            var edmModelFactory = ServiceLocator<IEdmModelFactory>();
            if (edmModelFactory == null)
            {
                return ServiceLocator<IEdmModel>();
            }
            else
            {
                return edmModelFactory.EdmModel(CurrentDataSource);
            }
        }

In the  Initialize method, which MVC calls automatically, we are going to set the Request’s IEdmModel.   We get the IEdmModel from an interface called IEdmModelFactory.

namespace ODataServices.Interfaces
{
    public interface IEdmModelFactory
    {
        IEdmModel EdmModel<TSource>(TSource source);
    }
}

We are missing error handing because it could return null, I’ll add that in later right now I’m just trying to get it up and running.  Since Entity Framework seems pretty popular with people that used WCF Data Services, plus that’s what we use, I’m going to add a default EdmModelFactory for Entity Framework DbContext.

namespace ODataServices
{
    //https://gist.github.com/dariusclay/8673940
    //http://stackoverflow.com/questions/22711496/entityframework-model-first-metadata-for-breezejs
    //https://gist.github.com/raghuramn/5864013
    public class DbContextEdmModel : IEdmModelFactory
    {
        private const string CsdlFileExtension = ".csdl";

        private const string CodeFirstContainer = "CodeFirst";

        private const string EntityConnectionMetadataPatternText =
            @"^(res://\*/(?<name>[^\|]+))(\|res://\*/(?<name>[^\|]+)?)*$";

        private const RegexOptions EntityConnectionMetadataRegexOptions =
            RegexOptions.Compiled | RegexOptions.CultureInvariant | RegexOptions.IgnoreCase |
            RegexOptions.ExplicitCapture;

        private readonly Regex _entityConnectionMetadataPattern = new Regex(EntityConnectionMetadataPatternText,
                                                                           EntityConnectionMetadataRegexOptions);

        private Stream GetCsdlStreamFromMetadata<TSource>(ObjectContext context)
        {
            var metadata = new EntityConnectionStringBuilder(context.Connection.ConnectionString).Metadata;
            var assembly = Assembly.GetAssembly(typeof(TSource));

            var csdlResource =
                _entityConnectionMetadataPattern.Matches(metadata)
                                                .Cast<Match>()
                                                .SelectMany(m => m.Groups["name"].Captures.OfType<Capture>())
                                                .Single(c => c.Value.EndsWith(CsdlFileExtension));
            return assembly.GetManifestResourceStream(csdlResource.Value);
        }

        private IEdmModel NotCodeFirstModel<TSource>(IObjectContextAdapter source)
        {
            using (var csdlStream = GetCsdlStreamFromMetadata<TSource>(source.ObjectContext))
            {
                using (var reader = XmlReader.Create(csdlStream))
                {
                    IEdmModel model;
                    IEnumerable<EdmError> errors;
                    if (!CsdlReader.TryParse(new[] { reader }, out model, out errors))
                    {
                        return null;
                    }
                    return model;
                }
            }
        }

        private IEdmModel CodeFistModel(DbContext context)
        {
            using (var stream = new MemoryStream())
            {
                using (var writer = XmlWriter.Create(stream))
                {
                    System.Data.Entity.Infrastructure.EdmxWriter.WriteEdmx(context, writer);
                    writer.Close();
                    stream.Seek(0, SeekOrigin.Begin);
                    using (var reader = XmlReader.Create(stream))
                    {
                        return EdmxReader.Parse(reader);
                    }
                }
            }
        }

        public virtual IEdmModel EdmModel<TSource>(TSource source)
        {
            var dbContext = source as DbContext;
            if (dbContext == null)
            {
                return null;
            }

            var objContext = (IObjectContextAdapter)source;
            if (objContext.ObjectContext.DefaultContainerName == CodeFirstContainer)
            {
                return CodeFistModel(dbContext);
            }
            else
            {
                return NotCodeFirstModel<TSource>(objContext);
            }

        }
    }

At the top of the code I listed where I got some of these methods and I haven’t tested it with a Code First Entity Framework yet, for testing I just setup a model first since it’s the easiest to get up and running.  So I can’t be sure that the container name for code first is CodeFirst, it’s on my list.  Also I don’t know if checking the container name is the best approach to know if a DbContext was generated code first or not.  This class doesn’t cache the IEdmModel and we will want to add that.  But I feel that is the classes responsibility and not the controllers.  That’s way the method EdmModel<TSource> is virtual.  We can inherit from this class and add a wrapper around our caching logic and call into base when we need a new IEdmModel – again about giving the developer the flexibility to determine when they want to cache or not.

Now to add the default implementation of IEdmModelFactory to the ODataServiceController.

        static ODataServicesController()
        {
            // Load up default implementations
            _serviceLocatorDefaults[typeof(IEdmModelFactory)] =
                new Lazy<object>(() => new DbContextEdmModel());
        }

Let’s first work on getting $metadata returned back

Update the ProcessRequest method.

        protected virtual HttpResponseMessage ProcessRequest()
        {
            var pathHandler = Request.GetODataPathHandler();
            var metadata = GetMetadata();
            var path = pathHandler.Parse(metadata, GetODataPath());
            Request.SetODataPath(path);
            Request.SetODataRouteName("OData");
            if (path.Segments.Any(s => s.SegmentKind == ODataSegmentKinds.Metadata))
            {
                var odataMediaTypeFormatter = new ODataMediaTypeFormatter(new DefaultODataDeserializerProvider(),
                                                                          new DefaultODataSerializerProvider(),
                                                                          new[] {ODataPayloadKind.MetadataDocument});

                var formatter = odataMediaTypeFormatter.GetPerRequestFormatterInstance(typeof (IEdmModel), Request,
                                                                                       new MediaTypeHeaderValue(
                                                                                           "application/xml"));
                var response = Request.CreateResponse();
                response.Content = new ObjectContent(typeof (IEdmModel), metadata, formatter);

                return response;
            }
            return new HttpResponseMessage(HttpStatusCode.NotImplemented);
        }

now in the ODServiceController pass the Get request to the ProcessRequest method.

    public class ODServiceController : ODataServicesController<SampleDataEntities>
    {
        public HttpResponseMessage Get()
        {
            return ProcessRequest();
        }
    }

With this done you should be able run in debug mode and go to localhost:yourport/api/odata/$metadata and get the xml document back.

Part 2 I’ll be working on query an EntitySet and Entity.

 

Tags: , ,

Monday, April 14th, 2014 OData No Comments