Prepare your environment

To be able to use the PnP Provisioning framework you need two key components. First of all you need to have Windows Management Framework v4.0 installed in order to generate the Cmdlet help. This can be downloaded from http://www.microsoft.com/en-us/download/details.aspx?id=40855 . Secondly you need to download the setup files and install the Powershell cmdlets. The latest release of the framework and instructions can be found here.

Once this is done you are ready to use the PnP Provisioning framework. To move our configuration to a different tenant/site we’ll create a template of our site and use it to configure the site on the new location.

Export the configuration to a template

Start your Powershell environment to access the newly installed cmdlets. The export process starts with connecting to the source environment where you configured your solution. Use the Connect-PnPOnline cmdlet and specify the url of your site.

You will be asked to enter your credentials.

Use Get-PnPProvisioningTemplate to create a XML that contains the description of your site.

In the -out parameter you can fill in the complete path of the XML-file. If no path is used, the file is created in the current directory of the Management Shell (c:\pnpDemo).

Now we have created an XML file that contains the configuration of your site. You will recognize the following nodes in the XML:

Sitefields

<pnp:SiteFields>
<Field … \>
<\pnp:SiteFields>

ContentTypes

<pnp:ContentTypes>
<pnp:ContentType>
<pnp:FieldRefs>
<pnp:FieldRef … \>
<\pnp:FieldRefs>
<\pnp:ContentType>
<\pnp:ContentTypes>

Lists

<pnp:Lists>
<pnp:ListInstance>
<pnp:ContentTypeBindings>
<\pnp:ContentTypeBindings>
<pnp:Views>
<\pnp:Views>
<pnp:FieldRefs>
<\pnp:FieldRefs>
<pnp:Security>
<\pnp:Security>
<\pnp:ListInstance>
<\pnp:Lists>

Security (site)

<pnp:Security>
<\pnp:Security>

Navigation

<pnp:Navigation>
<\pnp:Navigation>

Apply Template

After creating a new site we can apply our template to it. So first connect to the destination site

And apply the template

Our Approach

• You can also use other commands to customize a site, but we prefer the XML. This allows you to make changes and reapply it again. When it’s a new object it will be created otherwise it will be updated.
• We prefer to use several XML’s that just contains one type of object: Site Fields, Site content types, lists, …
• First try out your templates in a site on your development environment before doing it in production
• In some cases we also use other Pnp Commands. Look here for an overview.

Hopefully the above will help you!

Microsoft Flow is a cloud-based platform for business process automation, which allows an Office 365 tenant user to build elaborate workflows and integrate numerous sources using connectors to Office 365, Azure, third-party vendor platforms,  gateways to on premises servers, as well as custom connectors or HTTP requests to virtually any external service.

In this article we will focus on its ability to replace legacy SharePoint Designer workflows. As a proof of concept we migrate a  SharePoint Designer 2013 approval workflow to MS Flow and highlight the key differences and challenges.

The SP 2013 solution

The solution has the following main components

• a SharePoint custom list allowing end-users to create an expense request for approval to the management
• a Sharepoint Designer 2013 workflow, triggered by the “on create” event
• a SharePoint workflow task list, and two custom content types, “Approval task” with actions Approve/Reject/Rework and “Rework task” with actions Cancel/Resubmit
• a SharePoint custom list of “Executive approvers” who are the final approvers in the approval flow

A request  submitted  by a user contains general information about the request (title, ID, description, amount) as well as approvers designated by the requester (like his general manager, local unit manager…). Some of the designated approvers are optional.

Every request passes sequentially through up to 6 approvers before it gets approved.

• Whenever an approver rejects a request, the request gets the status “rejected” and the workflow ends.
• If all approvers approve an item, the request gets the status “approved” and the workflow ends.
• If one of the approvers requests a rework, the user has the ability to cancel his request or rework it. In the last case the approval starts all over again to approver 1.

The requester gets an e-mail confirming whether his request has been accepted or rejected and a task if his request needs rework. All tasks also trigger a mail alert by default to their assignee.

The main building block of the workflow is a loop which goes through all the approvers, assigns a task to each, waits for the result, and if the results is not “approved” it breaks the loop to step into the alternate stages.

The MS Flow solution

In MS Flow our solution contains these components

• a flow with the main business logic
• some variables
• connections to services, in our case SharePoint Online and Outlook Online
• actions on SharePoint Online and Outlook Online
• a trigger from SharePoint Online

Editing

Editing a workflow in MS Flow is simply a matter of going to the MS Flow app under the O365 app launcher and switching a flow to edit mode. No need to install an app or plugin, the environment is also fully compatible with all current browsers.

The user interface is slick and fluid, yet there are some limitations at this time, here are some we experienced:

• When you create a flow with a lot of switch/case or parallel conditions, the graphical representation becomes quite wide (zooming in/out with your browser can help), and it becomes cumbersome to move horizontally between portions of the flows, also the browser doesn’t always get it right when to display the scroll bar.
• Text fields (like mail body fields) trigger an interpretation of your keystrokes, and this can cause some delays, typos and/or mess up your text when mixing static and dynamic values. Also some fields support HTML encoding and some not, which is not clearly documented or indicated.
• There is no way to copy paste, both within a flow, nor between flows, you can however do a “save as” and continue working on a copy. Also you could in theory export a package and then copy a part of its code which is stored in a JSON file format.
• There is a lack of help or guidance when entering an expression, the documentation on this topic is rather basic, so you end up doing a mixture of searching on the web (check the links section) and some trial and error attempts.
• Specifically for SharePoint, when updating an item in a list you are forced to provide all data for required fields, there is no action to update a single field. Also the integration of MS Flow solutions in SharePoint is limited, in the modern experience the Flow features are simply links to the Flow environment, there is no integrated interface for managing or manually starting workflows like SP 2013 workflows.

For keeping an overview of a long workflow, a good practice is to group as much as possible actions together with “scopes” (grouping actions) with a distinctive name. There is also the possibility to comment your code in every action.

Some tips if you plan to work with SharePoint (custom) lists:

• When you make changes to the (list) content type, these changes are never directly reflected in MS Flow, not by re-selecting the list in the SP actions, not by switching edit mode on and off; you need to close your flow, then in the flow overview page (where you see the run history, connections, owners, …) go back to editing, you’ll see the changes.
• If you use a lookup field in a list where you want to update items, it will report a failure on update if you use “Title (linked to item)”; curiously the issue doesn’t occur when creating an item, and even while it reports a failure it actually successfully updates the item (but can cause your flow to stop). If you want to avoid the failure report use plain Title or any other field.

Triggers and overall logical buildup

The starting point for every workflow in MS Flow is a “trigger”, an action initiating the workflow, this can be any event coming from any connected service that you can subscribe to. Alternatively, this can be a schedule executing the flow on a defined frequency.

In our case we want to start the workflow whenever a new item is created in the SharePoint request list, but also when an item needs a rework.
In SP 2013 we would use the “on created” event, if the requester asks to resubmit, the workflow can simply go back to the initial step without the need to restart.
However MS Flow has no “go to” or “go back” so we need the be able to capture the “on update” event as well. So in the case of a resubmit, the request gets updated by the flow and a this triggers a new execution of the flow on the request.

Be aware that capturing a larger scope of events (both creates AND updates) has an impact on the quota of runs allowed in your MS Flow plan. As of writing of this article, the free plan of Flow that comes with O365 enterprise licenses allows up to 750 runs per user or 2000 per tenant. Above that amount of runs you’ll need a paid plan. Check the link section below for more information. Also note there is currently no way to detect the number of runs in the free plan (other than by noticing when a run fails, or by downloading the run history).

As we initially used the “Create” trigger, at some point we needed to change it into the “Create and update” trigger. In SP 2013 you can simply check or uncheck the different startup options (manual, on create, on update). A current limitation of MS Flow is that you can’t easily change a trigger. As soon as you are using data from your trigger in your workflow, your trigger is implicitly attached to your flow and it can’t be removed and replaced. The editing menu of your trigger will show this message:

The reason is: as other actions in your flow depend on its input, the flow would break. If your new trigger uses the same connection and you don’t want to recreate the full flow from scratch, there is a workaround:

• export the workflow as a zip package (option “create as new”)
• make sure to rename the original workflow to avoid conflicts when importing
• unzip the package file, look for the definition.json file
• replace the trigger by editing the file:
  in our case we replaced “When_an_item_is_created” with “When_an_item_is_created_or_modified” , “GetOnNewItems” with “GetOnUpdatedItems” and “onnewitems” with “onupdateditems”
• recreate the zipfile (make sure the Microsoft.Flow folder is in the root of the zip file
• import the package, the flow is now triggered by the new action

Variables

Since the user has the ability to edit his request during the flow (and our approval process can take up to 3 or 4 weeks), we first store all the request data into variables that we’ll use during the flow. This way we avoid that the data changes during the flow execution.

Also we can store some tenant related data like an url of the site collection in one place instead of in all the the connector actions that depend on it. If we move the flow later to a different environment (eg. from test to production) we only need to change a few variables.

Finally, storing the most important data in variables has the benefit of  having them suggested as first when entering dynamic data, so you don’t need to scroll or search in a lengthy list of data from all the connections and actions.

Approval

Approval in MS Flow is accomplished either by Approval Tasks or Approval Mails.

Approval Tasks reside in the MS Flow itself and are visible under the Approvals tab.

When you use the “Start an approval” action, it adds an entry in this list and generates a mail notification to the assignee.

There is however a significant limitation at the time of writing, you cannot customize the list of actions, it is limited to Approve and Reject.

So for our case, as we needed Approve/Reject/Rework and Resubmit/Cancle, we decided to use the “Send an Approval Mail” action, which sends a mail with the customized list of actions included as link button. When clicking on one of the buttons, the approver is directed to an Azure page which sends the selected action to the flow (once an action has been activated, you cannot trigger the same or another action).

There is a drawback and an advantage to this action: if you include all relevant data in the approval mail, the approver doesn’t need to leave his mail environment to asses the request and perform the appropriate action, this in replacement of the SP 2013 approval experience, where the user has to click on the link to the item, edit it, select an action and save the item (or click on an action button). Replacing all these steps with a single click is an appreciable improvement in terms of user experience.

The drawback is that you have no trace of the approval that are open, nor in Flow nor in SharePoint, so we ended up building a feature into our flow, which adds approval “reminders” in a custom list.We cannot include however the actions from the mail into this list, so the mail is the single point of failure for the process. In our case there are no external mailboxes or -environments involved so the risk is rather limited.

Tracing

There is no tracing feature builtin MS Flow, so you’ll have to use either an 0365 repository (custom list) to trace your actions from within Flow, or for a more enterprise solution store all traces in Azure Service Bus where they can be processed by any subscribing service.

User Account

A final note on the user account running the flow. You can have different accounts for the main flow and the connectors, so your flow identity and the one sending mail for instance can be different. It is however advised to use a service account for the execution of the flow, be aware that it will the necessary permissions to carry all the required actions in SharePoint and Outlook, as well as the licences to access those services.

Also if you plan to protect your user account with Multi Factor Authentication, there is no way to use an app password in Flow so you’ll end up having to refresh the authentication token on a frequent basis in the Connectors tab. If you have any connection failure frequently causing a flow to stop, MFA might be the cause.

Conclusion

With MS Flow now for over a year in GA release, we can conclude that it is a more than valid option to migrate your workflow to.

However, before moving an existing SharePoint workflow to this platform, we recommend to list your main business requirements (the “must haves”) and check these with a proof of concept. It will also help familiarize your customer with the way MS Flow works, since it is not part of the SharePoint stack, things like approval look different and and are managed out of the SharePoint environment, so it is better to make him aware upfront of the changes in the user experience.

Pros

In our  experience it looks more reliable and certainly faster than the SP 2013 workflow engine.

The learning curve is fairly shallow, with exception of the (advanced) expressions. The general buildup is and user interface is very intuitive. the community is now large enough to provide you will kinds of help.

There is virtually no limit to the possibilities of MS Flow when you start expanding in a platform like Azure functions, Web API, Azure automation, or extending it as a Logic Apps solution in Visual Studio.

With Connections (shared with another 0365 service called PowerApps) it is clearly becoming one of the key components of the 0365 environment, and “the way to go” for any workflow based solution in the Microsoft cloud.

Cons

The user interface has some issues especially for “long” and “wide” workflows, long text fields, it offers no ability to quickly copy paste or change your trigger …

The inability to group logic in steps and navigating up/down between these can be an issue for more complex workflows.

Custom tracing is (not yet) available.

The actions of the SharePoint connector have different (often more limited) possibilities than their SP Designer counterparts, always take that into consideration with a proof of concept; you can always workaround these limitations with SharePoint REST API calls or Azure functions but you’ll need to assess the complexity and effort of these workarounds. As the platform constantly evolves, it might be an unnecessary investment in a few months time.

Links

MS Flow pricing

https://emea.flow.microsoft.com/en-us/pricing/

O365 Power Users Forums

https://powerusers.microsoft.com

0365 UserVoice

https://office365.uservoice.com/

This year Spikes was represented by 12 “Spikees” at the European SharePoint Conference 2017 (#ESPC17) in Dublin, Ireland – or officially “European SharePoint, Office 365 and Azure Conference”. While we had a great mix of focus areas present, I mainly focussed on the IT Pro and business tracks. In this blogpost I will try to translate the highlights of the information I gathered over 3 days into a functional/business perspective.

Self service and digital transformation

“We still haven’t transformed, we’ve migrated.” @bniaulin at #ESPC17 pic.twitter.com/637cPtphp8

— European SharePoint (@EuropeanSP) November 14, 2017

The common thread throughout the entire conference contained 2 parts: a huge focus on Self Service and the Digital Transformation process almost every company still has to go through.

Self service

Why require intervention of the IT/support department when the business requests a collaboration space? Or when they want to involve external parties? Or need a solution for publishing transparent information to the organisation or a specific part of it?

All of the above (and many more) can be created by relying on self service. Of course, IT can play a role in helping provide the best corporate tools towards the business. It is also important that before “releasing” all of the corporate approved solutions, the exercise has to be made regarding governance, policies and (naming) conventions.

Digital transformation

A great statement during one of the key notes was “We still haven’t transformed, we’ve migrated!”. Basically, technology is evolving or has already evolved into self service, scalable, modular solutions, but the companies themselves need to be digitally transformed as well.

Apparently many companies do not understand that if they don’t provide the right solutions to their people/business, they will find a solution themselves. Often not a corporate governed solution without consistent data back ups and potentially unsafe with a great risk of data loss, also known as “Shadow IT“. And we all know the cost of data loss …

Now is the time to start your company’s digital transformation and avoid losing your best employees, gain a higher level of trust and data security and get IT and business to start working closer together. This will result into a secure modern workplace which contributes to productivity and employee happiness. Eventually, employee happiness will significantly increase customer satisfaction.

Spikes can help you deliver this secure modern workplace, involving business and IT from the start by generating awareness, helping understand each other and eventually help you build a digital, safe and modern workplace. https://www.spikes.be

Office 365 Groups

“Who thought that Office 365 Groups were actually the same as Outlook Groups?”

Almost 1600 attendees, including the amazing key note speaker Benjamin Niaulin, all raised their hand. This is actually how Microsoft presented Office 365 Groups until recently: a distribution list which initiates/configures other collaboration products to get your employees going in no-time. WRONG.

An Office 365 Group is an Active Directory security group – basically nothing more, nothing less. Moving forward, you can imagine it’s a burden for the IT/support department to create a communication channel, document workspace, task manager … manually, every single time a security group is requested. Setting up these collaborative services & products is being triggered by the creation of an Office 365 (Security) Group – that part hasn’t changed. But remember: it’s not and Outlook Group; it’s an AD Security Group which initiates other collaborative services & products.

It is clear that Office 365 Groups are here to stay!

Modern team sites

If you have been working with SharePoint for over a year or longer, it is impossible to imagine SharePoint without classic team sites/classic pages. They work fantastic for SharePoint gurus, hard core IT-ers and Power Users … not so much for business people. Microsoft therefore launched “Modern pages” which also are a part of the Modern team sites.

These Modern team sites have a clear focus on the business to build and self sustain these sites and pages from bottom up. Although the modern pages are fairly easy to set up and manage, it is a pain that certain web parts have disappeared, compared to the classic web parts. If you’re not a technical person, just be aware of the SPFx framework which can be used to build custom solutions and web part for. They even will be as responsive as the out-of-the-box ones. So in some situations, developers can still build custom “modern” web parts where the business or functional requirements don’t meet with the out-of-the-box solutions or web parts.

Sneak peak: “Groupify”: be able in the future to transform your classic team sites into a team site, linked to an Office 365 Group – easy membership management included!

Good to know: more and more Modern web parts are being released! We at Spikes have access to many Office 365 tenants. This means that if a web part has been released by Microsoft into their “First Release” network, we will look into it and make sure to test it thoroughly. Next, we provide our customers with proactive feedback on the use cases these web parts could be used in and their potential added value.

In case a Modern team site is linked to an Office 365 Groups (security groups; remember?) business users are also capable of managing the security setting with one or two mouse clicks: Site Owners or Site Members. In a nutshell: Modern team sites are much more focused on business usage and maintenance – big win!

It is clear that Modern team sites are here to stay!

Communication sites

Where Modern team sites mainly function as part of a collaborative space, sometimes all you need is some sort of a publication platform. This is where Communication sites come into the picture. Again, they are focused on business usage and maintenance where normally no interference of IT or support should be necessary.

A communication site knows many use cases, for example:

• Event site with announcements and photos
• Landing site for a portal/local intranet
• Public HR site with benefits
• Tire-change schedule/registrations
• Training overview with registration option
• …

Did I mention that all Communication sites and Modern team sites/pages are entirely responsive by default and look truly awesome on any device?

Such as the features for the Modern team sites and its pages, also new things are being released as we speak where as some are on the roadmap for the near future. I was able to capture 2 roadmap slides from one of the ESPC-sessions (sorry for the bad quality of the first one – I hope it’s just readable enough):

It is clear that Communication sites are here to stay!

Hub sites

People that have been working with SharePoint in the past know that many Information Architecture models contain a lot of site collections, sites and especially a tremendous amounts of sub-sites. Many companies suffered from this way of creating hierarchies on site levels, since users had a hard time to either manage or master the concepts of permission inheritance and assigning unique or breaking permissions.

Since any new Communication or Modern team site is by default a separate site collection, all of these “entities” exist on the same, flat level. There is currently no way to connect these site collections, unless by adding navigation links manually to each site collection and maintain it continuously.

This is where Hub sites provide you with a solution. This soon to be released new type of site will give your (selected group of) users the option to make a Hub site from an existing Communication or Modern team site. This will be the ‘landing site’ which will become an entrance to other, linked Communication or Modern team sites.

Even when the organisation structure changes, Hub sites cover this entirely through simply changing relations between sites and their respective Hub site. #awesomeness

Each of the existing Communication or Modern team sites is going to have the ability to be linked with any existing Hub site. Keep in mind that they can only be linked to maximum 1 Hub site, not multiple.

Next to a top navigation that will be pushed through to linked Team or Communication sites, also a specific theme will be inherited by these linked sites. This causes to be able to navigate fairly easy through multiple site collections which otherwise are not linked at all.

Although they are not released yet, it seems that Hub sites are going to have a great impact on the existing information architecture models of many corporate environments.

Summary

Wrapping up this blog post, I want to mention that in my upcoming blogpost I will be writing about what I’ve learned at #ESPC17 regarding Flow, Teams & Stream and a deep dive into the currently available (in First Release) and to be announced modern web parts.

Please stay tuned!

Pieter

As an IT project manager, I searched for an easy and generic approach to detect the reasons (whether financial, operational, cultural, …), as well to define the goals and the measurements needed to monitor the business case!

Although not a new concept, I believe that the “gains and pains”-concept – often used for defining value proposition – may be used as “a framework” that helps the writer of a business case to describe the reasons together with its goals for starting up a project

Putting theory into practice, I give an example from one of my customers, that  briefly describes how a “pain” was translated into a solution and how the goals were defined for that specific solution

First we discussed which pains my customer had. Soon it became very clear that involvement of people was a major pain, this due to historical reasons and the culture spread. From a technology perspective, we tried to help the customer in finding a solution to help the customer’s employees improving their involvement within their professional environment. The assumption was that employees could easily use any external social collaboration platform to spread their ideas about the company publicly.  Therefor a private social collaboration platform was introduced, allowing ALL employees to facilitate horizontal and bottom-up communication.

As a result, employees got more and more involved into the company processes, as they were able to express their thoughts about almost any process in the company,. The platform allowed upper management to capture their employees’ thoughts and comments as an input for improving processes and circumstances. Measuring involvement became easy: the more people who shared their ideas on the private social collaboration platform, the better processes were adapted and the better these processes were perceived by the employees.

This is what happened mathematically. By defining a pain, the company got a gain at the same price of one pain!

• Pain:  Lack of involvement
• Goal:  Improve involvement of employees and indirectly improve processes
• Gain:  Improved processes
• How:   A private social collaboration tool and capturing the ideas of employees about internal processes
• KPI and Measurements:  Counting the number of input and measuring the effect of adapted processes onto the employees via a simple tool called “Survey”!

Of course, we all agree that most business cases consist of multiple pains and gains, having each their own goals and measurement tools.

Hopefully this simple example may help other people in defining a business case.

The tool can be used by all O365 users, you do not need to be a global administrator to run the tool. Instead you only need read access to the source data and write access on the destination.

The tool is able to migrate smaller as well as large data batches and execute them in bulk.

Below you can find some more info on the Preview version of the new SharePoint Migration Tool from Microsoft.

The preview version can be downloaded on the following location.

Requirements

Before you begin, check the requirements prescribed by Microsoft.

Following table gives a short overview what happens with the permissions you had allocated to your source data

Using the SharePoint Migration Tool

When opening the SharePoint Migration tool, you will see the following screen.

When you first launch the SharePoint Migration tool, you will be prompted for your Office 365 username and password. The Office 365 credentials you provide will be where the files will be migrated.

After you successfully signed in the tool is asking you where you source data is located:

Before starting the tool, note down where your data is located and where you want to migrate those files.

As you can see above you will need to select the source location and enter the URL for the SharePoint Online site or OneDrive for Business site. Keep in mind you can only select data out of SharePoint 2013 sites or from a local file share or network path.

• SharePoint on-premises:
  If you select the SharePoint on-premises option, you will be asked to enter the name of the SharePoint Server site where your files are located and prompted for your credentials for that site. You will indicate what document library you wish to migrate.

• File Share:
  If you select the File share option, you will be asked to enter the location of the file share, the URL of the SharePoint Online site and document library where they will be moved.

For this demo purpose we will select the File Share option, procedure for the SharePoint option will be identical.

In the next popup screen select the folder or documents (Source) you want to migrate.

When clicking next, you will be able to fill in the destination URL. When filled in the requested detail just click on “Add”

In the next screen you will see the configuration you’ve made for source and destination. Within this screen you have the possibility to add more tasks or to export the tasks to CSV, which can be used later in a bulk migration process.

After you click Migrate, the progress of your migration jobs will be displayed. As they complete, you can view either detailed or summary reports of an individual job or view a single summary report that includes all migration jobs submitted during this session.

Migration report can be downloaded as well to check if the migration is executed correctly.

All folders, subfolders & files were migrated correctly into the SharePoint Online site.

Summary

First impression, The SharePoint Migration Tool lets you migrate your files from SharePoint on-premises document libraries or your on-premises file shares and easily move them to either SharePoint or OneDrive in Office 365.

The SharePoint Migration tool is an excellent tool for simple 1-to-1 data migrations. However, when there are more complex data migration requirements, 3th party tooling such as provided by AvePoint or ShareGate is still the best answer to successfully migrate content.

The following list illustrates the more complex needs:

• Migration of custom permissions
• Migration of 3th party workflows
• Migration of user alerts
• Mapping of site templates, permission levels, users & groups
• Migrating Managed Metadata
• Export from SharePoint to Csv
• Bulk Metadata edit
• Migrating Farm Architecture or Content DB’s
• Migrate sites to different templates or levels
• Version cleanup

Furthermore the new SharePoint Migration Tool currently only supports SharePoint 2013 on-premises & fileshares as a content source.

Ruben Plehiers

Business Innovation Consultant @ Spikes

In order to follow along with this post, it is important that you have an IoTIFY account. You can sign up for a free account here.

There are two things we need to know before we can start setting up our network simulations. First, the credentials of a device to connect to the Azure IoT Hub. Second, the actual message structure of the simulated device to be sent to the IoT Hub. Let’s begin with the first one.

Go to the Azure Portal and visit your IoT Hub. Click Device Explorer > Add. Fill in a unique ID for your device and save the settings (see figure below).

Once the device is setup, we can use the Device Twin Explorer to retrieve a SAS (Share Access Signature) token that will be used later to connect the device to the IoT Hub. In the management tab you have an overview of the different devices. Select your device and click the SAS Token button. A dialog appears where you again need to select the correct device as well as a Time-To-Live (TTL) for your token. Click the Generate button and your token will appear.

For the actual message structure, we will visit our Intel IoT gateway and go to the Node-RED environment. Double click the Create Payload element in the flow. Here we can see the actual structure of the message we are sending to our IoT Hub. Record this for future reference.

It is a simple structure with only three elements:

{ "deviceId": "client0", "timestamp": "08/18/2017 12:46 PM", "temperature": 36 }

Please note that the above step can be very different in your setup. The way described here assumes you have followed the previous posts as well.

Next, we can start creating our simulations using the IoTify service. Go to the IoTify website and login using your account. In your network tab, you should see something like the following:

Templates are the mechanism used by IoTify for simulating your device. Several samples have been provided to you. However, we will create two templates of our own. One is using the MQTT protocol and one is using the HTTP REST protocol.

MQTT Template

Create a new template and you should see something similar as the figure above. Next, fill in the blanks concerning MQTT parameters and credentials as follows:

1. Connection type: MQTT
2. Protocol: mqtts (TLS 1.2)
3. Endpoint: .azure-devices.net:8883 (replace with the name of your IoT Hub).
4. Client ID: (replace with the ID of the device created above)
5. Credentials:
   1. Username: .azure-devices.net/
   2. Password : Everything that comes after ‘SharedAccessSignature=’ in the SAS Token generated above (see underlining in the figure x).

In the “Specify Message Contents” section we will specify the topic endpoint as well as the message structure.

1. Topic: devices//messages/events/
2. Message Contents : A JavaScript function body returning the actual message. This is the same structure as the one determined above. The message basically contains a device id, a timestamp and a temperature. Numerous expressions can be used to specify this message. Here we opted for a timestamp representing current time and a randomly generated temperature between -10 and 45 degrees Celsius.

HTTP Template

Again create a new template and you should see something similar as the figure above. Next, fill in the blanks concerning the HTTP parameters as follows:

1. Connection type: HTTP
2. Protocol: https (TLS 1.2)
3. Host: .azure-devices.net:443 (replace with the name of your IoT Hub).
4. Path: /devices//messages/events?api-version=2017-06-30
5. Method: POST
6. Additional Header:
   1. Create a new Authorization header with as value everything that comes after ‘SharedAccessSignature=’ in the SAS Token generated above (see underlining in the figure x).

Note: In other blog posts, I have seen people ask about how to retrieve the actual API version. You could use the date that is provided in the documentation. At the time of writing this one says “2016-02-03”. I have retrieved mine from the source code of the Node.js SDK for connecting devices to the Azure IoT Hub (which is available on GitHub). This also worked for me!

In the “Specify Message Contents” section we will specify the message structure in the same way as the we did for the MQTT template.

Simulations

Next, we can start building our simulations. State the number of clients that you want to initialize as well as how many times you want them to repeat sending a message.

Additionally, you can apply a number of network parameters to make your simulation better resemble real-life network conditions e.g. average delay, jitter,… You can do all this using predefined templates as well.

One important thing to note about the simulations is that in the case of MQTT connections IoT Hub only supports one active MQTT connection per device. New MQTT connections using the same credentials will result in dropping the existing connection. In terms of performing simulations using the MQTT protocol this resulted in on average 50% success rate for our simulations (i.e. using 1000 clients through the same device credentials). Performing the same simulation using the HTTP protocol resulted in a 100% success rate.

Conclusion

In this blog post, we covered running simulations on our IoT solution, i.e. IoT Hub, using the IoTify service. We made the connection between IoTify and IoT Hub using the MQTT and HTTP protocol.

I’m creating some open source angular components to use across multiple projects within the organization. This might be useful for other people as well. The components will be open source and reside in a Github repo and will be packaged into an npm package.

The first component in this library will be a Treeview. At the core this is a pretty easy component to build, but further on I will tell you about a small bump on the road and the design decision that was made to bypass that bump.

Let’s get to it.

Treeview

I needed a treeview for a project with the following requirements:

• Recursive
• Tri-state select visualization
• Retrieve selected items on a level per level basis

spikes-treeview

In theory, we could do without a main component for a simple treeview. But if you would like to show Collapse/Expand All buttons, they would need to live outside of the treeview items.

Later, other functionalities can be added like e.g. search.

But I digress. This is the main tree-view component:

html

<div class="row"><div *ngIf="treeviewConfiguration.showToggleAll" class="col-lg-12"><button type="button" class="btn btn-secondary" (click)="toggleAll(true)"><i [ngClass]="[treeviewConfiguration.mainFont, treeviewConfiguration.expandIcon]"></i> Expand All</button><button type="button" class="btn btn-secondary" (click)="toggleAll(false)"><i [ngClass]="[treeviewConfiguration.mainFont, treeviewConfiguration.collapseIcon]"></i> Collapse All</button></div><spikes-treeview-item [treeviewItems]="treeviewItems" [treeviewConfiguration]="treeviewConfiguration"></spikes-treeview-item></div>

The main component exists out of 2 parts:

• Toggle All buttons:
  Through configuration there can be decided to show the Expand All/Collapse All The default setting is false.
• Spikes-treeview-item:
  Explained below.

ts

@Component({  selector: 'spikes-treeview',  templateUrl: './spikes-treeview.component.html',  styleUrls: ['./spikes-treeview.component.css']})export class SpikesTreeviewComponent implements OnInit {  private _treeviewConfiguration: models.ITreeviewConfiguration;  private _treeviewItems: Array<models.TreeviewItem> = [];@Input()  set treeviewConfiguration(config: models.ITreeviewConfiguration){this._treeviewConfiguration = new models.TreeviewConfiguration();Object.assign(this._treeviewConfiguration, models.TreeviewConfiguration.defaultTreeviewConfiguration, config);  }  get treeviewConfiguration(){if (this._treeviewConfiguration == null){  this._treeviewConfiguration = new models.TreeviewConfiguration();  Object.assign(this._treeviewConfiguration, models.TreeviewConfiguration.defaultTreeviewConfiguration);}return this._treeviewConfiguration;  }  @Input()  set treeviewItems(items: Array<models.TreeviewItem>) {this._treeviewItems = [...items];  };  get treeviewItems() {return this._treeviewItems;  }  constructor() { }  ngOnInit() { }}

The main component takes 2 inputs:

• treeviewConfiguration: configuration of the treeview (url github)
• treeviewItems: items to show

spikes-treeview-item

The treeview item is the recursive component of the treeview. It has a reference to itself as you can see in the html below.

html

<div class="row"><div class="col-lg-12"><ul><li class="spikes-treeview-item" *ngFor="let item of treeviewItems"><span *ngIf="item.childItems != null && item.childItems.length > 0"><a (click)="item.toggle()"><i [ngClass]="[treeviewConfiguration.mainFont, !item.expanded ? treeviewConfiguration.expandIcon : treeviewConfiguration.collapseIcon]"></i></a></span><span *ngIf="treeviewConfiguration.showSelection"><a (click)="item.setState(item.itemState, item.itemState === 3 ? 1 : 3)"><i [ngClass]="[treeviewConfiguration.mainFont, item.itemState === 1 ? treeviewConfiguration.checkedIcon : item.itemState === 3 ? treeviewConfiguration.uncheckedIcon : treeviewConfiguration.indeterminateIcon]"></i></a></span><span (click)="item.setState(item.itemState, item.itemState === 3 ? 1 : 3)">{{ item.displayText }}</span><div *ngIf="item.expanded"><spikes-treeview-item [treeviewItems]="item.childItems" [treeviewConfiguration]="treeviewConfiguration"></spikes-treeview-item></div></li></ul></div></div>

So what’s going on here:

<li class="spikes-treeview-item" *ngFor="let item of treeviewItems">

We loop through all the items passed to the spikes-treeview-item component.

<span *ngIf="item.childItems != null && item.childItems.length > 0">

wrapper for the expand/collapse icon for each item.
Hidden when the current treeview-item does not have any child items.

<a (click)="item.toggle()">

open or close the underlying child-items

<i [ngClass]="[treeviewConfiguration.mainFont, !item.expanded ? treeviewConfiguration.expandIcon : treeviewConfiguration.collapseIcon]">

an icon that is styled through css classes.
Default: font-awesome

<span *ngIf="treeviewConfiguration.showSelection">

wrapper for the checkbox icon for each item.
Through configuration we can indicate if we want to show checkboxes. The default is true.

<a (click)="item.setState(item.itemState, item.itemState === 3 ? 1 : 3)">

select or unselect the treeview-item.
Selecting (or unselecting) has an effect on the underlying child-items as well as on the parent items above. (Recursively)

<i [ngClass]="[treeviewConfiguration.mainFont, item.itemState === 1 ? treeviewConfiguration.checkedIcon : item.itemState === 3 ? treeviewConfiguration.uncheckedIcon : treeviewConfiguration.indeterminateIcon]">

an icon that is styled through css classes.
Default: font-awesome

<span (click)="item.setState(item.itemState, item.itemState === 3 ? 1 : 3)">

shows the displaytext of the treeview-item.
Has the same click response as the checkbox icon.

<div *ngIf="item.expanded">

when the current item has child-items and is expanded, the contents of this div are shown.

<spikes-treeview-item [treeviewItems]="item.childItems" [treeviewConfiguration]="treeviewConfiguration">

the child items of the current treeview-item.

ts

@Component({  selector: 'spikes-treeview-item',  templateUrl: './spikes-treeview-item.component.html',  styleUrls: ['./spikes-treeview-item.component.css']})export class SpikesTreeviewItemComponent implements OnInit {  private _treeviewConfiguration: models.ITreeviewConfiguration;  private _treeviewItems: Array<models.TreeviewItem> = [];  @Input()  set treeviewConfiguration(config: models.ITreeviewConfiguration){this._treeviewConfiguration = new models.TreeviewConfiguration();Object.assign(this._treeviewConfiguration, models.TreeviewConfiguration.defaultTreeviewConfiguration, config);  }  get treeviewConfiguration(){if (this._treeviewConfiguration == null){  this._treeviewConfiguration = new models.TreeviewConfiguration();  Object.assign(this._treeviewConfiguration, models.TreeviewConfiguration.defaultTreeviewConfiguration);}return this._treeviewConfiguration;  }  @Input()  set treeviewItems(items: Array<models.TreeviewItem>) {this._treeviewItems = [...items];this.normalizeTreeviewItemParents();  };  get treeviewItems() {return this._treeviewItems;  }  constructor() { }  ngOnInit() { }}

The item component takes the same 2 inputs:

• treeviewConfiguration: configuration of the treeview (url github)
• treeviewItems: items to show

You might notice that I’m not using an input tag for the checkbox. I tried using it at first, but then I hit a snag.

Tri-state checkbox

There is this property on a checkbox input called “indeterminate”, it cannot be set through HTML, only through script and can be used to indicate that a checkbox is in an “indeterminate” state. You see this kind of thing often when you have a treeview with checkboxes where a parent is “indeterminate” when one or more of its child items, but not all of them, are checked.

At first, I wanted to make use of this property on the checkbox, but I soon found out that browsers handle a click on an indeterminate checkbox differently. This is the small bump that I referred to earlier.

Chrome :

Edge :

As you can see, in the gifs above, a checkbox that is checked and set to indeterminate gets either unchecked (Chrome) or checked (Edge) after clicking it.
Because of this, I changed this to using i-tag and setting an appropriate class to it. I used font-awesome as a default font set for setting the icons. But I left the option to replace them through configuration.

  Get selected items

I added a helper class that lets you retrieve the selected items from the treeview:

getTreeviewSelectedItems(treeviewItems)

• Input:
  the treeviewItems of the treeview
• Output:
  a flat list of all selected items, without any information about the level on which the item resides.

getTreeviewSelectedItemsPerLevel(treeviewItems)

• Input:
  the treeviewItems of the treeview
• Output:
  a list of TreeviewLevel Each level has an array of selected items. The level is a number indicator starting from level 1.

Full code and demo app

The code for this component is available on Github under src/components/spikes-treeview.
A demo app showing this component is also available.

Feel free to comment below!

With the new HttpClient introduced in Angular 4.3.1, I think it’s a good moment to write a little update.

I’m following the same setup as the previous post:

• Angular project
• Web Api project
• Windows Authentication

Let’s get started.

Web Api

Little has changed for the Web Api part. The short version is:

• application.config

  <authentication><anonymousAuthentication enabled="true" userName="" /><basicAuthentication enabled="false" /><clientCertificateMappingAuthentication enabled="false" /><digestAuthentication enabled="false" /><iisClientCertificateMappingAuthentication enabled="false"></iisClientCertificateMappingAuthentication><windowsAuthentication enabled="true"><providers><add value="Negotiate" /><add value="NTLM" /></providers></windowsAuthentication></authentication>

• Web API
  • Add CORS
  • Configure CORS

    var cors = new EnableCorsAttribute("http://localhost:30033", "*", "*") { SupportsCredentials = true };

  • Configure for json instead of XML

    config.Formatters.JsonFormatter.SupportedMediaTypes.Add(new MediaTypeHeaderValue("text/html"));

  • Web.config settings

    <authentication mode="Windows"/><authorization>  <allow users="?"/></authorization>

You can find the in-detail version here.

Some points of interest in the Web API setup:

• In the CORS configuration we need to set which url’s are can contact our api. I’m developing an angular project with the angular-cli. The default is http://localhost:4200

var cors = new EnableCorsAttribute("http://localhost:30033,http://localhost:4200", "*", "*") { SupportsCredentials = true };

• I’ve noticed that while setting up a project, as described in my previous post, works as expected if you start from scratch. Starting from the github repo of my sample, sometimes you keep getting a 401 unauthorized on the post request.
  If that’s the case, you can easily solve this by selecting the web api project in visual studio and open up the properties. Make sure to set Anonymous Authentication to Enabled.
  

Setting up our Angular application

I’m using the angular-cli. If you don’t know how to use it, check out their getting started information before continuing.

After setting up the application, we include the new HttpClientModule in the app.module:

import { HttpClientModule } from '@angular/common/http';@NgModule({  declarations: [AppComponent  ],  imports: [BrowserModule,HttpClientModule  ],  providers: [  ],  bootstrap: [AppComponent]})export class AppModule { }

Note: take special note of the fact that I’m using the new http import from @angular/common/http and not the old(er) one from @angular/http which still exists and works as before. I want to make use of newer functionality so I’m going for the new http implementation.

Next, add 2 services to the application.

• AuthenticationService

  getUser(): Observable<string> {console.log('Calling getUser');let serviceUrl: string = `${environment.serviceBaseUrl}auth/getuser`;return this.http.get(serviceUrl, {responseType: 'text'})  .map((rslt: string) =>{return rslt;  });  }

• DataServices

  save(data: models.IPostDataModel): Observable<string> {console.log('Calling getUser');let serviceUrl: string = `${environment.serviceBaseUrl}data/save`;return this.http.post(serviceUrl, data, {responseType: 'text'})  .map((rslt: string) => {return rslt;  });  }

Note: the new http client uses json by default. And tries to parse what is returned by a request as json. In our sample web api we return a string, so we need to specify another responseType. Which in our case will be text. More about this can be found here.

Import and provide those services in the app module :

import * as svcs from './services/';providers: [svcs.AuthenticationServiceService,svcs.DataServiceService],

Create the necessary data model for our server side PostData object.

export interface IPostDataModel {Id: number;Name: string;IsTrue: boolean;CreatedOn: Date;}export class PostDataModel implements IPostDataModel {public Id: number;public Name: string;public IsTrue: boolean;public CreatedOn: Date;constructor(obj?: IPostDataModel) {if (obj != null) {Object.assign(this, obj);}}}

 
I’m using the AppComponent for demonstrating the windows authentication by adding 2 buttons to it.

<div style="text-align:center">  <div class="row"><div class="col-lg-12"><button type="button" (click)="testAuthentication()">Authentication</button><pre style="border:solid 1px;min-height:21px;" [ngClass]="[authBack]">{{authRslt}}</pre></div><div class="col-lg-12"><button type="button" (click)="testPostData()">Post Data</button><pre style="border:solid 1px;min-height:21px;" [ngClass]="[postBack]">{{postRslt}}</pre></div>  </div></div>

And of course the functions that will be executed when clicking the buttons.

  testAuthentication(): void {this.authSvc.getUser()  .subscribe(r => {this.authRslt = r; this.authBack = 'success';},e => {console.log(e); this.authBack = 'error';}  );  }  testPostData(): void {this.dataSvc.save(new models.PostDataModel({Id: 1, Name: 'DeBiese', IsTrue: false, CreatedOn: new Date()}))  .subscribe(r => {this.postRslt = r; this.postBack = 'success';},e => {console.log(e); this.postBack = 'error';}  );  }

HttpInterceptor

In the new http client module, intercepting a request and/or response is very easy. Writing an interceptor is like writing an angular service that implements the HttpInterceptor interface.

An example of an interceptor is the following:

import { Injectable } from '@angular/core';import {  HttpRequest,  HttpHandler,  HttpInterceptor,  HttpEvent} from '@angular/common/http';import { Observable } from 'rxjs/Observable';@Injectable()export class WinAuthInterceptor implements HttpInterceptor{  constructor(){}  intercept(req: HttpRequest<any>, next: HttpHandler): Observable<HttpEvent<any>> {return next.handle(req);  }}

This interceptor is doing nothing now. It intercepts a request and passes it on without change.

In our case, we want to make sure that every request adds the current user’s windows credentials. The intercept method changes to this:

  intercept(req: HttpRequest<any>, next: HttpHandler): Observable<HttpEvent<any>> {req = req.clone({  withCredentials: true});return next.handle(req);  }

The request (response) object needs to be immutable, so we need to clone the original request before we return it. We set the withCredentials parameter to true so that every outgoing request adds those credentials.

Lastly, we need to make sure that this interceptor is being used. So we need to register it in the app module.

  providers: [{  provide: HTTP_INTERCEPTORS,  useClass: int.WinAuthInterceptor,  multi: true},svcs.AuthenticationServiceService,svcs.DataServiceService  ],

Testing the application

Let’s fire up both the Web API and the angular application:

After clicking the buttons:

Both requests are successful, meaning the windows authentication is working the way we want it to work.

Conclusion

Using windows authentication with the new HttpClientModule in Angular 4.3.1 (or higher) is fairly easy. Just write an interceptor and make sure it is being used by providing it in your app module.

I hope this post provides you with enough information to set this up yourself.

The full code base of this test project can be found on github.

Feel free to comment below!

But what happens when you follow all the steps correctly (at least you think so) and the response you get is “Unauthorized”.
Well, that’s what happened to me. I poured blood, sweat, and tears into finding the solution.

The following steps helped me to troubleshoot the issues I had:

Ping-Pong

First, I created a “Ping-Pong” (Hello World) service. I even created one with and one without authentication. By doing this I could be sure all the other possible issues would be ignored.

Chrome Dev Tools & jwt.io

Of course, I also used the Chrome developer tools to debug the JavaScript code calling the service. Here I could find the bearer token and decrypt it using https://jwt.io/ which was giving me a lot more useful information:

• Finding the token:

• Decrypting the token: 

The properties that I thought were helpful were:
– aud –> This should correspond with the “resource” parameter in the web.config
– unique_name –> The owner of this token
– scp –> The permissions granted by this token. Example: AllSites.Write

App Service Editor

Validating the web.config settings of your webAPI in Azure can be done easily using the App Service Editor. You can be sure validating the correct settings by opening this file from Azure.

Always validate that the “tenant” and “audience” parameters are configured correctly.

Custom Authentication Provider

I created a custom Authentication Provider by inheriting from the OAuthBearerAuthenticationProvider class. This code will not do anything except making debugging a lot easier.

public class OAuthBearerAuthenticationProviderEx : OAuthBearerAuthenticationProvider{public override Task ValidateIdentity(OAuthValidateIdentityContext context){return base.ValidateIdentity(context);}public override Task RequestToken(OAuthRequestTokenContext context){return base.RequestToken(context);}}

To implement this extra class I had to update the startup class of my webAPI.

public class Startup{public void Configuration(IAppBuilder app){HttpConfiguration config = new HttpConfiguration();WebApiConfig.Register(config);ConfigureAuth(app);app.UseWebApi(config);var jsonFormatter = config.Formatters.OfType<JsonMediaTypeFormatter>().First();jsonFormatter.SerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver();}public void ConfigureAuth(IAppBuilder app){app.UseWindowsAzureActiveDirectoryBearerAuthentication(new WindowsAzureActiveDirectoryBearerAuthenticationOptions{Tenant = ConfigurationManager.AppSettings["ida:Tenant"],TokenValidationParameters = new TokenValidationParameters{ValidAudience = ConfigurationManager.AppSettings["ida:Audience"]}, Provider = new OAuthBearerAuthenticationProviderEx()});}}

Diagnostic settings

Finally, I changed the diagnostic settings for more debug information. You can do this by adding the following to your web.config file:

<system.diagnostics><switches><add name="Microsoft.Owin" value="Verbose" /></switches></system.diagnostics>

 As the result of this change you will get a lot more information in the output window such as a detailed error message as described below.

In the previous example you can see the message “Audiences did not match”. This was caused by a difference in the web.config settings and the configuration in the token. Figuring this out without the detailed information could be very frustrating.

Summary:

• Hello world call
• Developer tools
• Decrypting the bearer token
• App service editor to validate the files in Azure
• Extending the OAuthBearerAuthenticationProvider class
• Enable verbose logging

I hope one of these steps helps you to solve your issue.

Rob Lempens

–