Tuesday, November 26, 2013

Consuming The SharePoint 2013 REST API From Managed Code

Chris Domino, Director, Enterprise Architect


As I get deeper and deeper into CSOM for SharePoint 2013, I've begun to discover some things that just don't work. That's not to say that they aren't supported by the client library or that they error out; they just silently fail and do nothing. Naturally, as with virtually every SharePoint 2013 CSOM issue I've ever had, I always first look into this being a security issue or an App Model misconfiguration.

But the setting of my findings of these issues wasn't through the App Model; it was actually a farm solution that used "conventional" CSOM to communicate with a remote web application. When I refer to "conventional" CSOM, I mean C# code using Microsoft.SharePoint.Client that just so happens to be running on a server. It's not SharePoint Server Object Model code; rather it's part of a job, a service, or a console or mobile app that's talking to a remote farm...or even the local farm.

Since the entry point into our REST API discussion will be within the paradigm of C# CSOM code, I want to begin by showing the logic that news up a "conventional" ClientContext, since we don't have TokenHelper sitting atop the security mountain of the App Model to do all the black box initialization work for us:

Code Listing 1

  1. public static ClientContext GetContextFromWebDotConfig()
  2. {
  3. //initialization
  4. ClientContext context = new ClientContext(ConfigurationManager.AppSettings["CSOM_URL"]);
  5. //get csom credentials
  6. context.Credentials= new NetworkCredential(
  7. ConfigurationManager.AppSettings["CSOM_User"],
  8. ConfigurationManager.AppSettings["CSOM_Password"],
  9. ConfigurationManager.AppSettings["CSOM_Domain"]);
  10. //no timeout (doesn't quite seem to work)
  11. context.RequestTimeout= int.MaxValue;
  12. //return
  13. return context;
  14. }

The four entries in the web.config file explicitly point to the destination SharePoint site collection (CSOM_URL) and authenticate as a well-known Windows user. This is effectively equivalent to SPSecurity.RunWithElevatedPrivileges, except better, as we can control exactly who we're impersonating.

Anyway, the point is that I can't blame the App Model (because I'm not using it) and I can't blame security (because I can impersonate anyone I want, which in this case is God). It took a while, but I eventually summoned up the audacity to assume that it wasn't my code that was failing; rather, it was CSOM itself that was letting me down.

So I turned my attention to the REST API, hoping that being a "lower-level" service (and thus allowing me to completely customize the request to SharePoint) I would have the power to do what I needed. I operate under the assumption that everything is possible via one of the SharePoint APIs; if one fails, another must pick up the slack. So far, these failings include:

  • Deleting a web
  • Setting "EnableFolderCreation" on a list
  • Setting "ContentTypesEnabled" on a list

It turns out that there's not a lot out there on the web discussing how to interact with the SharePoint 2013 REST API from "server" C# code. The technology is tailor-made to work with AJAX and the SharePoint JavaScript Object Model, but what about using it from the server to talk to remote farms? Or from a service, since we're not supposed to have anything from the Server Object Model in our apps?

Using HttpClient seemed like a good approach, especially if asynchronous support was a requirement. However, you have to deal with a lot of HTTP junk to get it to work. Therefore, I decided to go a level deeper down to the HttpRequest itself so that I can control the call entirely. If CSOM is failing me, I want to get as close to the wire as possible.

The RESTer

I created a class called "The RESTer" (or just RESTer.cs) that wraps all the HTTP requests needed to interact with the SharePoint 2013 REST API. This baby performs the following: manages headers, handles authentication, and provides support for both conventional CSOM and App Model CSOM. This is an important distinction, as the two paradigms for SharePoint development have vastly different security approaches.

The Payload

At the heart of this beast is the payload: the actual raw JSON that SharePoint reads off the wire. Each REST call is a single operation against a SharePoint object: create a list item, set a property on a web, delete a document from a library, etc. Therefore, this payload is essentially a JSON representation of the object we're operating against, and metadata about what we're doing to it.

The first bit of RESTer code we'll look at is a method that abstracts the creation of the payload by taking in two parameters: the "type" of object we're operating against, and a dictionary of properties. This dictionary's name/value pairs correspond to properties on the target object, and their vales. And as we're used to with RESTful programming, the verb of the request specifies what kind of CRUD operation we're doing.

Code Listing 2

  1. private string BuildPayload(ObjectType objectType, Dictionary<string, string> properties)
  2. {
  3. //initialization
  4. string type = string.Empty;
  5. //determine object type
  6. switch (objectType)
  7. {
  8. //list
  9. case ObjectType.List:
  10. type = "SP.List";
  11. break;
  12. }
  13. //build post body
  14. StringBuilder sb = new StringBuilder();
  15. properties.Keys.ToList().ForEach(k => sb.AppendFormat("'{0}': {1}, ", k, properties[k]));
  16. //return
  17. return string.Format("{{ '__metadata': {{ 'type': '{0}' }}, {1} }}", type, sb.Remove(sb.Length - 2, 2).ToString());
  18. }

In Line #1, ObjectType is a simple enum that represents the type of object we'll be operating against. Right now, I only use list, but this can easily be expanded to support any SharePoint JavaScript objects. The complete set is here. The switch statement on Line #6 converts the enum value into the text expected by the REST API. The properties are "serialized" into JSON on Line #15; no need for anything fancy here.

The magic happens on Line #18. Here is where we build out the body of REST calls that POST data to the server. In conventional MVC Web API, clients POST or PUT a JSON object over the wire, where it is automatically desterilized into a POCO on the other side. But as we all know, SharePoint is never quite conventional, so we have to build this metadata object instead. An example of JSON that updates the title of a list looks like this:

string json = "{ '__metadata': { 'type': 'SP.List' }, 'Title': 'New List Title' }";

The Authentication

Authentication in general makes a new twist in SharePoint 2013 with the App Model stuff. But at the end of the day, CSOM, weather it's running on a server or as part of an app, is a pretty bow on an ugly WCF package; we're still very much in the business of auth-ing service calls. Believe it or not, the App Model security is less complicated than the conventional CSOM, so let's discuss that part of the RESTer next.


Configuring a SharePoint 2013 environment to work with apps is outside the scope of this post. I swear I'll get to it soon!


The reason things are harder in this paradigm verses the App Model is because the aforementioned TokenHelper class that's generated for you every time you create a new SharePoint 2013 App project in Visual Studio isn't available. Now don't get me wrong: configuring everything on your environment to make TokenHelper work is a nightmare. However, assuming we're starting from an App-ready farm, conventional CSOM calling into the REST API needs some extra love.

Without getting into all the gory OAuth details, SharePoint 2013 REST API calls are authenticated via an access token that is generated for each user's "session" with the server. What TokenHelper is kind enough to do is provide this for us, using a certificate and a plethora of configuration behind it. This is the core of App Model security. So first, let's see the code to initialize the RESTer from an app, and then we'll look at the conventional paradigm.

Code Listing 3

  1. public RESTer(Uri siteUrl, string accessToken)
  2. {
  3. //set members
  4. this.AccessToken = accessToken;
  5. this.SiteUrl = this.ParseSiteUrl(siteUrl);
  6. }

This is the "App Model" constructor for the RESTer. It takes in the url of the target web site, as well as the access token provided by a properly-configured App Model CSOM ClientContext. We saw earlier how we get our conventional context; to extract an access token from an app's client context, refer to the second code listing in Part 2 of my App Model series. Then pass this string to the aforementioned constructor.

Since our conventional CSOM won't have an access token, we need to go fetch one. To start that process, we need to use the second RESTer constructor. It takes in a web url all the same as the first one, but instead of an access token, it starts its security journey with an instance of ICredentials. Here's what that looks like:

Code Listing 4

  1. public RESTer(Uri siteUrl, ICredentials credentials)
  2. {
  3. //set members
  4. this.Credentials = credentials;
  5. this.SiteUrl = this.ParseSiteUrl(siteUrl);
  6. //get digest
  7. this.Digest = this.GetDigest();
  8. }

We'll get to Line #7's GetDigest method in a second. It first needs to be stated that when newing up a RESTer with this constructor, simply pass in the Credentials property from your already-established ClientContext. Notice that both constructors scrub the url parameter (which is a Uri) though "PraseSiteUrl." All this does is convert it to a string and trim off any trailing slashes. This is important because we still be in the unfortunate business of cobbling fragments of urls together to form our RESTful calls.

The much more interesting GetDigest method makes an introductory call into the REST API to get our access token. This process is not too far off in nature to an NTLM handshake: make a call to the server with our credentials, get a request form digest (essentially an access token) back, and use that to auth any other calls we make during this session. The following code does this work:

Code Listing 5

  1. private string GetDigest()
  2. {
  3. //initialization
  4. XmlDocument doc = new XmlDocument();
  5. //build request
  6. HttpWebRequest request = (HttpWebRequest)WebRequest.Create(string.Format("{0}/_api/contextinfo", this.SiteUrl));
  7. request.Method = Method.POST.ToString();
  8. request.ContentLength = 0;
  9. //set credentials
  10. if (this.Credentials != null)
  11. request.Credentials = this.Credentials;
  12. else
  13. request.UseDefaultCredentials = true;
  14. //make the call
  15. using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
  16. {
  17. //get response
  18. using (StreamReader sr = new StreamReader(response.GetResponseStream()))
  19. {
  20. //load xml
  21. doc.LoadXml(sr.ReadToEnd());
  22. XmlNamespaceManager manager = new XmlNamespaceManager(doc.NameTable);
  23. manager.AddNamespace("d", "http://schemas.microsoft.com/ado/2007/08/dataservices");
  24. //return
  25. return doc.SelectSingleNode("/d:GetContextWebInformation/d:FormDigestValue", manager).InnerText;
  26. }
  27. }
  28. }

Starting on Line #6, we assemble a fairly straightforward credentialed HttpWebRequest that will post an empty body to the "contextinfo" endpoint. This article does an excellent job of describing the details. This line also feeds you a taste of the url building we'll be doing throughout this class. It might seem hacky, but welding urls together is required of any functionality that wraps a RESTful API. Later we'll break down the convention of what SharePoint 2013 expects its REST API urls to look like.

Next, Line #'s 7 and 8 configure our empty post. "Method" is another enum that will be included in the code attached to this article that simply contains all the HTTP verbs we'll be using. Next, we authorize the request on Line #'s 10 - 13, passing along the credentials from our CSOM ClientContext. The next few lines do the minutia of a WebRequest, so let's skip down to Line #21 where we start to extract our request form digest.

From here, the rest of the method is devoted to loading the XML that comes back from the request, parsing it, and getting our access token from it. Line #'s 22 and 23 deal with the namespace junk required for us to be able to query against it. Finally, Line #25 executes the query that grabs the "FormDigestValue" node's inner text. This return value is then stored in a global variable that the RESTer will use to do its thing.

The Request

Now that we have our payload, and can convince the server to talk to us, let's put all the pieces together into our REST call. For this example, we'll be setting two properties on a list. The following code listing is the bread and butter of the RESTer: it assembles the request object, along with its headers, url, authentication, and payload. Let's look at the code first, and then break it down step by step:

Code Listing 6

  1. private string DoRequest(Method method, Scope scope, string operation, string operationParameters, string payload)
  2. {
  3. //initialization
  4. string middlePart = string.E