How to gather data from SharePoint using REST API on external website

Recently I was struggling to gather in an external web application data from SharePoint. I was not able to use the App Model as the customer didn’t have the proper App Infrastructure in place. In my example I wanted to call from an external web application the SharePoint REST API to gather search results. This is possible when using a service account, but I didn’t want that approach, because I wanted security trimmed search results. Passing through credentials is impossible as the environment was using NTLM authentication and then you’ll face the Double Hop problem. Activating Kerberos would solve this issue, because that features delegation, but was also not feasible as you need to modify the farm configuration which was not allowed. Next idea was to do it Client Side!

Client Side means in our case using Javascript. The browser should perform the request to SharePoint and then process the results to display it in an external web application. The advantage is that the end users credentials are used to authenticate against SharePoint and that security trimmed results are retrieved. Starting with this ideas I immediately struggled with browser security restrictions. The web application is running on a different domain then SharePoint and then you’ll face cross-domain issues. There are a few approaches to solve this:
– CORS (Cross-origin resource sharing)
– JSONP (JSON with padding)
– IFrame with PostMessage API

CORS is quite easy to implement in combination with JQuery. However it is not usable with SharePoint, because you need to change the server configuration. CORS works only when the server sends modified HTTP Access-Control-Allow-Origin headers. This requires server modifications and is therefore not a solution in our case.

JSONP is an alternative for doing web service requests. Instead of doing a normal JSON request you load the webservice request as a script with a callback method which gets the data as argument. This technology is however not usable in combination with SharePoint. SharePoint sends with all responses a HTTP header X-Content-Type-Options: nosniff. Because of browser security improvements an additional check is performed by the browser to see if the MIME-type is correct. SharePoint doesn’t send the MIME-type which is allowed in script tages so the browser will block the script. There is an article on MSDN about this security measures in Internet Explorer. Conclusion is that this technique is also not usable in our case.

Another option is to use IFrames with PostMessage. PostMessage is an API introduced as part of HTML5. Most browsers support this (IE since IE8). The idea is to use an IFrame in the web application and open a SharePoint page in the IFrame. The SharePoint page contains some Javascript to call the REST API and then sends back the results using PostMessage. When using IFrames you’ll need to tackle two problems. First thing is that you need to make sure that both the web application and SharePoint is using HTTP or HTTPS. When combining them this will give security warnings in the browser. Another problem is that SharePoint sends a HTTP header standard which blocks it from using SharePoint content in IFrames. Luckily there is a solution for this. You need to include the following control on your aspx page to allow frames:

<% Register Tagprefix="WebPartPages" Namespace="Microsoft.SharePoint.WebPartPages" Assembly="Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<WebPartPages:AllowFraming runat="server"/>

This control when added to the page will remove the X-FRAME-OPTIONS HTTP header which prevents displaying SharePoint content in a IFrame/Frame. Be aware that this only works on ASPX pages! Otherwise you’ll still need server modifications by configuring IIS. The ASPX page can for example be uploaded to the style library of a Site Collection. This library has normally read permissions for All Authenticated users which is typically useful for scenario’s like this.

In the following sample I will show some code samples to make it possible to send information cross-domain using the SharePoint REST API. First step is create a web application outside SharePoint or just an HTML page with some Javascript. Add an IFrame to it with the URL to the helper page we will create later on and add the JQuery script library. Then use the following code to make sure that we can receive messages from the helper page in SharePoint:

$(document).ready(function () {
  if (typeof window.addEventListener !== "undefined") {
      window.addEventListener("message", receiveMessage, false);
  } else if (typeof window.attachEvent !== "undefined") {
      window.attachEvent("onmessage", receiveMessage);
   }
});

function receiveMessage(event) {
    var eventData;
    try {
       eventData = JSON.parse(event.data);
        // Implement your logic here!
    } catch (error) {
        // Implement some error handling here!
    }
}

In the above sample we are using JQuery to wait before the page is loaded. Then we add a eventreceiver to the window object to receive messages using the PostMessage API. In case a message is received a method called ReceiveMessage is called. There we need to parse the JSON message to an Javascript object and then we can implement our logic.

Now we have the application page in place we need to create the helper page which needs to be uploaded to SharePoint. Create a new aspx page (you can change a standard publishing page in SharePoint), add the AllowFramingControl to it and add a script reference to a Javascript file we will upload as well. In the new Javascript file we need to add the following code:

var origin = "http://urltowebapplication.com" // Make sure that origin matches the url of you web application with iframe!!!!

// Start executing rest call when ready
$(document).ready(function() {
  callRestAPI();
});

function callRestAPI() {
    // Construct rest api call
    var restUrl = _spPageContextInfo.webAbsoluteUrl + "/_api/web";
    $.ajax(
    {
       url: restUrl,
        method: "GET",
        headers:
        {
           "accept": "application/json;odata=verbose",
        },
       success: onSuccess,
       error: onError
    });
}

function onSuccess(data) {
    if(data)
    {
        var response =
        {
            header: "response",
            message: data
        };
        parent.postMessage(JSON.stringify(response ), origin);
    }
}

function onError(err) {
   var response =
   {
       header: "error",
       message: err
   };
   parent.postMessage(JSON.stringify(response), origin);
}

The above code sample again uses JQuery to wait before the page is loaded. Then it constructs a url to call the SharePoint REST API to retrieve details about the Web object. You can extend this by passing parameters by querystring, etc to make it more dynamic. After that it calls the REST API using an Ajax call. It uses some headers to make sure that JSON is returned instead of XML. Then the postmessage call is used to send the message to the parent in JSON format when the call was succesfull or not. On the other side the response can be checked to see what kind of data has been send by checking the header. This concept can be extended with things like two way communication using Postmessage. The disadvantage of all of this is that everything runs in the browser and that automatic logon in the browser needs to be supported. Otherwise instead of the helper page an access denied page will be loaded, resulting in no messages being send. You can’t detect this properly from code because access to the DOM in IFrames is blocked in case of cross-domain content. The only thing you can do is implement some timeout mechanism and send always a lifesign signal using Postmessage that the page has been loaded to let the other page know that everything is ok.

17 thoughts on “How to gather data from SharePoint using REST API on external website

  1. Kristina Pereyra

    This worked great for me, thanks. Here is a sample ASPX page (save into SiteAssets library) that will work on any announcements list.

    <%-- The markup and script in the following Content element will be placed in the of the page --%>

    $(document).ready(function() {
    GetAnnouncements();
    });

    function GetAnnouncements()
    {
    var listName = "Announcements";
    var titleField = "Title";
    var descField = "Body";
    var filterField = "Expires";

    var today = new Date();
    var dd = today.getDate();
    var mm = today.getMonth()+1; //January is 0
    var yyyy = today.getFullYear();
    var filterValue = mm+'/'+dd+'/'+yyyy;;

    var query = "/_api/Web/Lists/GetByTitle('"+listName+"')/items?$select=" +titleField+ "," +descField + "&$filter=" +filterField+ " ge '" +filterValue + "'";

    var call = $.ajax({
    url: _spPageContextInfo.webAbsoluteUrl + query,
    type: "GET",
    dataType: "json",
    headers: {
    Accept: "application/json;odata=verbose"
    }
    });
    call.done( (function (data, textStatus, err){
    out = "";
    for (itemIndex=0; itemIndex<data.d.results.length; itemIndex++) {
    var title = data.d.results[itemIndex][titleField];
    var desc = data.d.results[itemIndex][descField];

    if (desc==null) desc = '';

    var thisItemData = ''+title+': '+''+desc+''
    out += thisItemData;
    $("#Announcement").prepend(out);
    }
    }));
    call.fail(function (err,textStatus,errorThrown){
    //console.log(err);
    alert("Error retrieving Items: " + err.responseText);
    $("#resultsDiv").append(err.statusText)
    });

    }

    <%-- The markup and script in the following Content element will be placed in the of the page --%>

    Announcements

    Reply
    1. Chelsea Xue

      Hi Kristina Pereyra:

      I’m trying to do the same thing but I’m new to SharePoint development. so It is driving me crazy to get the result from the page. Do you have a more complete version of you code please? Then that would help a lot. I want to tried to compare so that to know where I’m missing. Or do you have any steps or advice that would help a lot too!
      Thanks a lot, in advance

      Best,
      Chelsea

      Reply
  2. Clint Schauff

    Hey Joreon. This is by far the best article I’ve seen on how to do this. Unfortunately, it doesn’t solve my problem since since my non-sharepoint site is HTTP and my sharepoint site is HTTPS. If you have any other suggestions, they would be welcome.

    Reply
    1. captain007 Post author

      Hi Clint,

      This is unfortunately not possible because then you’ve mixed content where the browser will block postmessage calls. It is a bad idea to mix up HTTPS and HTTP, because then you can leak information from a secure connection to a non-secure connection and is security wise blocked. Only solution is to make the other application use HTTPS. To test the principle you can temporary use a self-signed certificate in the non SharePoint site to test if it works with both HTTPS enabled.

      Reply
  3. Mario

    Hello Jeroen,

    Tried you solution but we are stuck with the error message: “Unknown server tag ‘WebPartPages:AllowFraming’ ”
    Can you advise?
    We use SharePoint 2010 foundation.

    Kind regards,
    Mario

    Reply
    1. captain007 Post author

      Mario, I don’t know if SharePoint foundation does support this control. Did you have enabled Publishing on your site? Could be that that needs to be enabled first.

      Reply
  4. Chelsea Xue

    Hi Jeroen,

    This post is so cool. Recently I’m doing some integration of another web application with sharepoint, and may need to use this method to get files details in sharepoint. But I’m new to sharepoint, there is one concept of
    “which needs to be uploaded to SharePoint. Create a new aspx page (you can change a standard publishing page in SharePoint)”
    is not clear to me.
    Can you give a direction of how to do it? or is there are any reference that I need to see so that I can upload my page to sharepoint?

    Thanks

    Reply
    1. captain007 Post author

      I suggest that you’ll first dig into how publishing and pages work within SharePoint. When using publishing, a Pages library is available on your site where you can create, download and upload pages. Using this concept you can modify the page to perform the changes needed as demonstrated in this post.

      Reply
      1. Chelsea Xue

        Hi, thanks for replying me….it’s driving me crazy to upload my aspx page in to site asset. it just doesn’t show the result. below are my default.aspx code. do you have any thoughts on it?

        var origin = “http://urltowebapplication.com”

        $(document).ready(function () {
        GetDocuments();
        });

        function GetDocuments() {

        var folder = “/sites/CRMDynamics/DYDev/Lists/contact/Abid%20Mian_0F04EBF2B93EE51180C6005056951757”;
        //var titleField = “Title”;
        //var descField = “Body”;
        //var filterField = “Expires”;

        //var today = new Date();
        //var dd = today.getDate();
        //var mm = today.getMonth() + 1; //January is 0
        //var yyyy = today.getFullYear();
        //var filterValue = mm + ‘/’ + dd + ‘/’ + yyyy;;

        var query = “/_api/Web/GetFolderByServerRelativeUrl(‘” + folder + “‘)/Files?$select = name, timecreated, timelastmodified, serverrelativeurl”;

        var call = $.ajax({
        url: _spPageContextInfo.webAbsoluteUrl + query,
        type: “GET”,
        dataType: “json”,
        headers: {
        Accept: “application/json;odata=verbose”
        }
        });
        call.done((function (data, textStatus, err) {
        out = “Test!!”;
        for (itemIndex = 0; itemIndex < data.d.results.length; itemIndex++) {
        var name = data.d.results[itemIndex][name];
        var timecreated = data.d.results[itemIndex][timecreated];

        if (desc == null) desc = '';

        var thisItemData = " " + name + "; " + timecreated + " "
        out += thisItemData;

        }
        $("#Files").prepend(out);
        }));
        call.fail(function (err, textStatus, errorThrown) {
        //console.log(err);
        alert("Error retrieving Items: " + err.responseText);
        $("#resultsDiv").append(err.statusText)
        });

        }

        What am I possiblely missing here?

        Thank you very much!

        Reply
        1. captain007 Post author

          I don’t see any place where the data is actually transferred outside the IFrame using the origin variable. Make sure the data is passed using postmessage to the other application. Also make sure the origin url is set properly, otherwise the browser will block the request. Test with a Javascript debugger attached to see where thing go wrong. First start to make the data retrieval happening in SharePoint itself, then start to expand it with postmessage. This makes it a lot easier to solve each problem one by one.

          Reply
          1. Chelsea Xue

            Hi Thanks, Actually I didn’t doing develop on external app yet. This is just the page that I uploaded into sharepoint and it shows blank instead of the data I wanted to see. It is a little mess when pasting the code here below is the complete .aspx page that I uploaded into sharepoint.

            Thank you very much!

  5. Chelsea Xue

    $(document).ready(function () {
    GetDocuments();
    });

    function GetDocuments() {

    var folder = “/sites/CRMDynamics/DYDev/Lists/contact/Abid%20Mian_0F04EBF2B93EE51180C6005056951757”;

    var query = “/_api/Web/GetFolderByServerRelativeUrl(‘” + folder + “‘)/Files?$select = name, timecreated, timelastmodified, serverrelativeurl”;

    var call = $.ajax({
    url: _spPageContextInfo.webAbsoluteUrl + query,
    type: “GET”,
    dataType: “json”,
    headers: {
    Accept: “application/json;odata=verbose”
    }
    });
    call.done((function (data, textStatus, err) {
    out = “Test!!”;
    for (itemIndex = 0; itemIndex < data.d.results.length; itemIndex++) {
    var name = data.d.results[itemIndex][name];
    var timecreated = data.d.results[itemIndex][timecreated];

    if (desc == null) desc = '';

    var thisItemData = " " + name + "; " + timecreated + " "
    out += thisItemData;

    }
    $("#Files").prepend(out);
    }));
    call.fail(function (err, textStatus, errorThrown) {
    //console.log(err);
    alert("Error retrieving Items: " + err.responseText);
    $("#resultsDiv").append(err.statusText)
    });

    }

    Reply
    1. captain007 Post author

      Where does it break? I suggest that you start using a script debugger (F12 toolbar in browser) and try to find out where it breaks and what the exception message is.

      Reply
  6. Pingback: You Can Use SharePoint Information Across Web Applications

  7. DaveyBoy

    I came to this page thinking it might bethe solution to my problems. Unfortunately it won’t work in Sharepoint Online (Office 365) because code blocks aren’t allowed in aspx files. If you have another solution, please share, but I expect you’ve moved on 🙂

    Reply
    1. captain007 Post author

      I did indeed moved on to something else in Web Development. I expect it can still work on Office 365. Did you try it without using the register assembly tag on the ASPX helper page?

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *