Tag Archives: Javascript

Using Select2 and SharePoint 2013 REST API with large lists and infinite scroll

Recently during a project I was involved with creating a custom application with JavaScript inside SharePoint. One of the requirements was to load data from a large list (> 5000 items) into a Select2 dropdown control. Select2 is a very handy dropdown control to use in custom applications. It features out of the box search functionality, infinite scroll and many more nice features. However I struggled a few days with some specific problems in combination with SharePoint. In this article I’ll describe how to use Select2 in combination with the SharePoint REST API and large lists.

Select2 uses out of the box a HTML select or input control to bind itself to. Because we need to specify a custom ajax function, Select2 will only work in combination with an input control which is hidden. Add to your page an input type hidden control, like this:

<input id="selectData" type="hidden" />

We need to fulfill a few requirements which are important:
– Able to select from a very large list
– Support searching on title
– Fast
– Reliable

To meet these requirements we need to overcome a few challenges. The first challenge is large list support. To overcome this issue, you need at least indexed columns in case you want to query on specific columns. Be aware that you need to set the indexes before filling the list with data! Second challenge is how to use Select2 with the SharePoint REST API. Documentation on the internet is very rare on this specific subject and also there are a few problems with OData and SharePoint 2013 which you’ll face. In the following sample I will show how to bind Select2 and make it possible to search and use infinite scroll. It works also very fast and reliable. The sample uses a list named Books and uses the Title field to filter on.

var webAbsoluteUrl = _spPageContextInfo.webAbsoluteUrl;

$('#selectData').select2({
    placeholder: 'Select a book title',
    id: function (data) { return data.Title; },
    ajax: {
        url: webAbsoluteUrl + "/_api/web/lists/GetByTitle('Books')/items",
        dataType: "json",
        data: function (term, page, context) {
            var query = "";
            if (term) {
                query = "substringof('" + term + "', Title)";
            }
            if (!context) {
                context = "";
            }
            return {
                "$filter": query,
                 "$top": 100,
                 "$skiptoken": "Paged=TRUE&amp;p_ID=" + context
            };
        },
        results: function (data, page) {
            var context = "";
            var more = false;
            if (data.d.__next) {
                context = getURLParam("p_ID", data.d.__next);
                more = true;
            }
            return { results: data.d.results, more: more, context: context };
        },
        params: {
            contentType: "application/json;odata=verbose",
            headers: {
                "accept": "application/json;odata=verbose"
            }
        }
    },
    formatResult: function (item) {
        return item.Title;
    },
    formatSelection: function (item) {
        return item.Title;
    },
    width: '100%'
});

// Function to get parameter from url
function getURLParam(name, url) {
    // get query string part of url into its own variable
    url = decodeURIComponent(url);
    var query_string = url.split("?");

    // make array of all name/value pairs in query string
    var params = query_string[1].split("&");

    // loop through the parameters
    var i = 0;
    while (i < params.length) {
        // compare param name against arg passed in
        var param_item = params[i].split("=");
        if (param_item[0] == name) {
            // if they match, return the value
            return param_item[1];
        }
        i++;
    }
    return "";
}

First thing we do is getting the current web URL. This is needed to construct the path to the REST API. Then we bind the Select2 control using JQuery. We define first a placeholder. This is the text that will be shown when nothing is selected. We also specify a function to determine the id. Select2 standard can’t handle the data format which comes back from SharePoint, so we need to set manually that the identifier is the Title field of the book. The same applies for the formatResult and formatSelection which makes sure that the title is shown as field in the dropdow list. The most important part is the ajax parameter. First a URL to the REST API is specified, then we specify dataType JSON, because we want all the data as JSON and not XML. We also need to set this explicitly in the contentType and headers of the HTTP calls we need to make. In the data parameter we specify the logic to handle the queries and make sure that the skiptoken is specified. Because of a bug/designed feature in the SharePoint REST API, the skip parameter doesn’t work. The linked article describes an alternative by using the skiptoken. We store this skiptoken partially in the context variable to make sure Select2 tracks it for each next call. The next important part is the results function. This function parses the result and grabs the p_ID value to store in the context variable. In case the __next value is set, we know that more data is available, so then we also set the more variable to true. Select2 uses the more variable to indicate if a next page with more items is available for infinite scroll.

How to gather data from SharePoint using REST API on external website

Recently I was struggling to gather in an external web application data from SharePoint. I was not able to use the App Model as the customer didn’t have the proper App Infrastructure in place. In my example I wanted to call from an external web application the SharePoint REST API to gather search results. This is possible when using a service account, but I didn’t want that approach, because I wanted security trimmed search results. Passing through credentials is impossible as the environment was using NTLM authentication and then you’ll face the Double Hop problem. Activating Kerberos would solve this issue, because that features delegation, but was also not feasible as you need to modify the farm configuration which was not allowed. Next idea was to do it Client Side!

Client Side means in our case using Javascript. The browser should perform the request to SharePoint and then process the results to display it in an external web application. The advantage is that the end users credentials are used to authenticate against SharePoint and that security trimmed results are retrieved. Starting with this ideas I immediately struggled with browser security restrictions. The web application is running on a different domain then SharePoint and then you’ll face cross-domain issues. There are a few approaches to solve this:
– CORS (Cross-origin resource sharing)
– JSONP (JSON with padding)
– IFrame with PostMessage API

CORS is quite easy to implement in combination with JQuery. However it is not usable with SharePoint, because you need to change the server configuration. CORS works only when the server sends modified HTTP Access-Control-Allow-Origin headers. This requires server modifications and is therefore not a solution in our case.

JSONP is an alternative for doing web service requests. Instead of doing a normal JSON request you load the webservice request as a script with a callback method which gets the data as argument. This technology is however not usable in combination with SharePoint. SharePoint sends with all responses a HTTP header X-Content-Type-Options: nosniff. Because of browser security improvements an additional check is performed by the browser to see if the MIME-type is correct. SharePoint doesn’t send the MIME-type which is allowed in script tages so the browser will block the script. There is an article on MSDN about this security measures in Internet Explorer. Conclusion is that this technique is also not usable in our case.

Another option is to use IFrames with PostMessage. PostMessage is an API introduced as part of HTML5. Most browsers support this (IE since IE8). The idea is to use an IFrame in the web application and open a SharePoint page in the IFrame. The SharePoint page contains some Javascript to call the REST API and then sends back the results using PostMessage. When using IFrames you’ll need to tackle two problems. First thing is that you need to make sure that both the web application and SharePoint is using HTTP or HTTPS. When combining them this will give security warnings in the browser. Another problem is that SharePoint sends a HTTP header standard which blocks it from using SharePoint content in IFrames. Luckily there is a solution for this. You need to include the following control on your aspx page to allow frames:

<% Register Tagprefix="WebPartPages" Namespace="Microsoft.SharePoint.WebPartPages" Assembly="Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<WebPartPages:AllowFraming runat="server"/>

This control when added to the page will remove the X-FRAME-OPTIONS HTTP header which prevents displaying SharePoint content in a IFrame/Frame. Be aware that this only works on ASPX pages! Otherwise you’ll still need server modifications by configuring IIS. The ASPX page can for example be uploaded to the style library of a Site Collection. This library has normally read permissions for All Authenticated users which is typically useful for scenario’s like this.

In the following sample I will show some code samples to make it possible to send information cross-domain using the SharePoint REST API. First step is create a web application outside SharePoint or just an HTML page with some Javascript. Add an IFrame to it with the URL to the helper page we will create later on and add the JQuery script library. Then use the following code to make sure that we can receive messages from the helper page in SharePoint:

$(document).ready(function () {
  if (typeof window.addEventListener !== "undefined") {
      window.addEventListener("message", receiveMessage, false);
  } else if (typeof window.attachEvent !== "undefined") {
      window.attachEvent("onmessage", receiveMessage);
   }
});

function receiveMessage(event) {
    var eventData;
    try {
       eventData = JSON.parse(event.data);
        // Implement your logic here!
    } catch (error) {
        // Implement some error handling here!
    }
}

In the above sample we are using JQuery to wait before the page is loaded. Then we add a eventreceiver to the window object to receive messages using the PostMessage API. In case a message is received a method called ReceiveMessage is called. There we need to parse the JSON message to an Javascript object and then we can implement our logic.

Now we have the application page in place we need to create the helper page which needs to be uploaded to SharePoint. Create a new aspx page (you can change a standard publishing page in SharePoint), add the AllowFramingControl to it and add a script reference to a Javascript file we will upload as well. In the new Javascript file we need to add the following code:

var origin = "http://urltowebapplication.com" // Make sure that origin matches the url of you web application with iframe!!!!

// Start executing rest call when ready
$(document).ready(function() {
  callRestAPI();
});

function callRestAPI() {
    // Construct rest api call
    var restUrl = _spPageContextInfo.webAbsoluteUrl + "/_api/web";
    $.ajax(
    {
       url: restUrl,
        method: "GET",
        headers:
        {
           "accept": "application/json;odata=verbose",
        },
       success: onSuccess,
       error: onError
    });
}

function onSuccess(data) {
    if(data)
    {
        var response =
        {
            header: "response",
            message: data
        };
        parent.postMessage(JSON.stringify(response ), origin);
    }
}

function onError(err) {
   var response =
   {
       header: "error",
       message: err
   };
   parent.postMessage(JSON.stringify(response), origin);
}

The above code sample again uses JQuery to wait before the page is loaded. Then it constructs a url to call the SharePoint REST API to retrieve details about the Web object. You can extend this by passing parameters by querystring, etc to make it more dynamic. After that it calls the REST API using an Ajax call. It uses some headers to make sure that JSON is returned instead of XML. Then the postmessage call is used to send the message to the parent in JSON format when the call was succesfull or not. On the other side the response can be checked to see what kind of data has been send by checking the header. This concept can be extended with things like two way communication using Postmessage. The disadvantage of all of this is that everything runs in the browser and that automatic logon in the browser needs to be supported. Otherwise instead of the helper page an access denied page will be loaded, resulting in no messages being send. You can’t detect this properly from code because access to the DOM in IFrames is blocked in case of cross-domain content. The only thing you can do is implement some timeout mechanism and send always a lifesign signal using Postmessage that the page has been loaded to let the other page know that everything is ok.

SharePoint 2013 Lazy loading Javascript

In SharePoint 2013 the Javascript loading mechanism seems to be changed a bit. When porting a SharePoint 2010 solution to 2013 I found out that sometimes some weird script errors where occuring when calling SharePoint Javascript libraries. On some pages in SharePoint 2013 it happens that not all SharePoint Javascript libraries are loaded because of the built-in lazy loading mechanism. This reduces bandwidth when loading pages, because no unneeded libraries are downloaded to the client. But this causes issues when you want to use not loaded libraries. The following sample Javascript codes shows how you can load some Javascript Libraries and then automatically call your function where you want to use those libraries:

//Register SOD's
SP.SOD.registerSod('core.js', '\u002f_layouts\u002fcore.js');
SP.SOD.executeFunc('core.js', false, function(){});;
SP.SOD.registerSod('sp.js', '\u002f_layouts\u002fsp.js');
SP.SOD.executeFunc('sp.js', false, function(){});
SP.SOD.registerSod('sp.core.js', '\u002f_layouts\u002fsp.core.js');
SP.SOD.executeFunc('sp.core.js', false, function(){});

function doSomething() {
   //Your Logic here which calls sp core libraries
}

// Load asynchronous all needed libraries
ExecuteOrDelayUntilScriptLoaded(function() { ExecuteOrDelayUntilScriptLoaded(doSomething, 'sp.core.js') }, 'sp.js');

In the example above we’re using the SP.SOD library provided with SharePoint. Those existed already in SharePoint 2010 and are still present in 2013. With the SOD library it is possible to lazy load Javascript files and load them on the moment you need them. The sample script exists of three parts. In the first step we register Sod’s (Script on Demand) where we define a key and as value the relative path to the Javascript file. We also call executeFunc to load the file using a dummy function. In the second step we create a custom function. This is the function where you want to call specific methods in the Javascript libraries loaded. Then we call ExecuteOrDelayUntilScriptLoaded. Because in this sample we want both sp.core.js and sp.js loaded, we nest it with another call to ExecuteOrDelayUntilScriptLoaded and finally let the callback call the function which needs the libraries loaded before executing. This method of loading scripts seems to work well in SharePoint 2013 and can also be used for other OOTB Libraries, like the Taxonomy Javascript Library. When your site is still running in SharePoint 2010 mode however, this doesn’t work properly. The registering of Sod’s seems to break with the 2010 way of loading the OOTB Javascript files so there you need only the ExecuteOrDelayUntilScriptLoaded calls. If you need to detect the mode in Javascript you can use the SP.Site.get_compatibilityLevel() function to retrieve that info using JSOM and then dynamically decide which method of loading to use.