SharePoint and I go way back.  I have a special place in my heart for SharePoint.  I got started with the product back in 2002 and it’s taken me places.  It brought me to the USA when I transferred with Microsoft to join the Windows SharePoint Services engineering team in 2007 and onto the Technical Product Management team in 2009.  I owe it a great deal.  Its for this reason that although my new day job isn’t directly tied to SharePoint any longer that I care about its future.  I lived its glory days in 2009-2012.  SharePoint fever was rampant. SharePoint Conferences sold out at 12,000+ people.  There was a gold rush in the enterprise collaboration space orbiting SharePoint.

I want SharePoint to rise again as a fantastic platform for enterprise collaboration.

I make no bones about the fact that I haven’t been SharePoint’s greatest proponent recently.  I feel that Microsoft was chasing shiny things at the expense of what made SharePoint great to begin with.  Things felt 80% done in it’s Enterprise Content Management and Collaboration capabilities.  Like a theme park that was great for a while but started to age and you couldn’t help but think what could have been if only it had been updated.  The SharePoint team went off and started building other things vs. focusing on updating its core features like Team Sites and document collaboration.  Competitors like Dropbox showed what a better experience could be like as SkyDrive (now OneDrive for Business) fell further behind.

All this flailing about drove me nuts.  I got disheartened honestly at Microsoft rewarding engineering for shipping new things that no one wanted while letting their golden goose slowly die.

There is nothing quite like a leadership change to signal to everyone that the status quo isn’t working.

Over a number of years SharePoint fell under a number of leaders who really didn’t understand the enterprise software in my humble opinion.  They came from the consumer side of Microsoft.  SharePoint got little attention.  (PS: who lets a multibillion dollar business stagnate the wither like that anyway?!!? nuts!)

Last year Jeff Teper returned to lead the SharePoint Product Group.  Many are hailing this as a turning point.  The return of the king in many ways.  After all … he is the father of SharePoint.

Jeff round 2…

Today we saw the first step in Jeff’s new era of SharePoint.  Jeff has put the first stake in the ground for his new vision for SharePoint and where he wants to take it.

…  empower people, teams and organizations to intelligently discover, share and collaborate on content from anywhere and on any device.

It focusses on four key areas:

• File sharing on any device
• Intranet/Team sites on any device
• New extensibility options for developers
• Enterprise features around security, compliance and privacy.

What I LOVE about this that it focusses on core SharePoint capabilities.  It’s radically updating what is already there vs. adding something new.

Someone finally got the memo that to stay relevant you need to focus on what made you great to begin with, don’t lose sight of your fans and incrementally build from a solid base.

SharePoint lost it’s way … the real question is … is all this too late?

I don’t think so.

Many think SharePoint’s game is over. There are a multitude of smaller more nimble productivity apps out there that do a lot of what SharePoint introduced to businesses back in 2001.  But the one thing many forget is Microsoft’s ability to execute end to end when the stars align just right.  Building the features is just the first star to align.  Company vision is another, Marketing is another, Sales is another.  When all these align at Microsoft it is an unstoppable force of nature.  None of SharePoint’s competitors can touch it when these things all come together end to end.  But it hinges on it being a core imperative at the company to make it happen.  I feel there is a very good chance of that given Satya personally pushed asked Jeff Teper to lead the SharePoint group again.  He understands how critical owning File sync and Share is, how crucial owning team collaboration is and how important personal productivity is.

Here’s Satya talking about SharePoint and OneDrive at today’s event

New product experiences…

Like I already mentioned, the first star needed for perfect alignment is product.  It all starts with compelling product capabilities.

Team sites vNext

Today the SharePoint team showed new product capabilities primarily in the team and document collaboration space with fresh web and mobile experiences.  This in my view is playing catchup for the last few years of rot in the SharePoint UX.

Honestly … it looks great and end users who are not used to existing SharePoint sites will love it.  Users already used to SharePoint will likely be confused about where the ribbon interface went and have a hard time finding things in the place they were used to.

This new UI is not particularly innovative or over and above what competitors have already, but it is super important table stakes that were desperately needed.

There is a new SharePoint mobile app.  To be honest, I am pretty ho hum about this.  In my experience companies give their intranet a brand.  Microsoft does with its intranet, MSW (MS Web).  I think SharePoint missed a great opportunity to give companies the ability to white label this app and deploy “MSW” to peoples phones, brand it with their colors and customize it with additional apps and capabilities.

Finding sites…

Almost since the dawn of time one of the issues with SharePoint was the rampant proliferation of SharePoint sites with little to no ability to find any of them easily.  SharePoint are revamping the Sites experience in O365 to show you all the sites you belong to.

Nice improvement!

Flow and Power Apps

This demands a whole post on its own!

SharePoint forms and workflows are still left wondering why they are not getting the attention they deserve.

Flow is a new tool (in preview) to help you build easy lightweight workflows that connect to different systems and tools.  It actually shows quite a bit of promise in my view, but it’s built by the Azure team and you can tell they are used to building things for developers, not end users.  Of course this will likely get better over time.  To really succeed it needs to be deeply rooted in the Office and SharePoint user experiences, a little bit of which is seen in the screenshot below with the “Add Flow” button in SharePoint.

This is one area where either Power Apps + Flow really has to grow up fast, or SharePoint needs to take the reigns of these tools.

Extensibility …

Ahhh … a topic near and dear to my heart.  Prior to leaving MS I ran the technical marketing team in Office 365 that focused on extensibility and the partner product ecosystem around Office 365 and it is still something I deeply care about.

Microsoft has always been a partner lead company.  Steve Jobs famously said the thing he admired most about Microsoft was it’s partner lead attitude.  Extensibility and the ability to build on and beside products is critical to a vibrant and healthy product ecosystem.

SharePoint didn’t get to a $2.5B+ business for Microsoft by being a great out of the box application.  It got there by partner companies building amazing tools, add-ins and products that enhance SharePoint with new capabilities.

SharePoint has had an ongoing identity crisis with extensibility.  Each new version introducing enough of a change to piss off developers.  Although on the one hand these new capabilities are great … developers also hate to have to change development processes and relearn best practices.

They are going to need to do that again this time around.  The fantastic part about it is of course the ability to surface UI directly in SharePoint using modern web development tools and frameworks.

I have some pretty bias views on this.  For a very long time there have been people bending SharePoint UI to do unnatural things.  They inject script and insert, tweak and change things to make the SharePoint UI do what they want.  They were driven to this by customer demands and consultants inability to say no when it comes to $.

This new framework for SharePoint UI gives developers a new UI API.  Documented hooks and points at which extending the UI is condoned and encouraged and something the team will support.  This should be the UI nirvana developers are looking for.  But the proof will be in the pudding and we will see if it goes far enough with its ability to deliver what customers are demanding.  There will be gaps, I’m 100% sure of that … but the key will be SharePoint’s ability to address those, driven in a collaborative manner with the community.

I have some concerns about the new framework with regards to its suitability in building commercial products.  I need to do more digging before I draw conclusions, but from what I know now it is very much targeted at enterprise developers, not ISVs.  The packaging and deployment model for example is different from what we have with Add-Ins today and I feel many of the benefits of the Add-In model that ISVs had, for example end user deployment and code isolation,  are not present in this new framework. I can only hope that a re-focus on these in the not too distant future.

Where to from here…

I feel we are seeing the first glimpse of the stars aligning again in SharePointlandia:

• I am impressed with the teams renewed focus on the core of what are SharePoint’s strengths and that feels good!
• Having this line in the sand is great … they need to do this twice a year and keep the pressure on catching back up to the pack.  They need to do this so that the shiny new things make sense again because the core is strong. You can’t innovate the body of a car if you leave the engine to languish (ala DeLorean).
• I am bullish that Jeff understands this and has rallied the team around this as a core priority.
• I am upset that they have not yet decided this is a core priority for ISVs building products with SharePoint as this is a critical component of what made them great the first time around.

I am also extremely happy to see some focus, care and attention on SharePoint again.

I want SharePoint to rise again.

-CJ

You can hear more about the #FutureofSharePoint in an interview on my podcast the Microsoft Cloud Show here.

One of the ways AC and I track how many people are listening to the Microsoft Cloud Show episodes we put out is by using the logs created for Azure Storage.  These track the various requests for the mp3 files for each episode.  You have to turn on this logging for your account and once this is done then log files are written into the /$logs/blob folder of your storage account.  You can read more about Azure Storage Analytics here: https://msdn.microsoft.com/en-us/library/hh343270.aspx

However, the way the storage logs are filed does not exactly make analyzing them easy in something like Excel.   They are logged in /year/month/day/hour/5min chunk folders/files in CSV format. (detail of what is logged is here).

Until recently our process for downloading the analyzing these was rather laborious and included:

1. Download all the logs files.
2. Use some PowerShell to squash all the thousands of CSV files into one big one.
3. Use SQL Import tool to take the CSV and import each line into a temp table in SQL Azure
4. Run a T-SQL script over the data to remove all the rows we were not interested in (we only wanted the logs for downloaded MP3s, not other assets etc.…)
5. Mutate the data in various columns for better reporting. e.g. pulling important parts from the User Agent string and pulling an episode number out of the filenames.
6. Finally inserting all the new rows into the final reporting Table
7. Using Excel to report over the data

It was error prone, painfully slow at times and required all these things to be done each time we wanted new statistics.  At times we would go months without doing this and were flying blind on how many people were listening.  Not really very good.

What we really needed was an automated process!

Azure Function Apps to the rescue!

In March 2016 Microsoft announced Azure Functions.  They are lightweight bits of code that can be run in Azure when certain events occur, like on a timed basis, when an HTTP request is made and when a blob is added to storage.  What is nice about them is that you are not paying for a Virtual Machine to sit there mostly idle.  You only pay for what you use.  You can write them in a variety of languages and trigger them in various ways.  If you know how to write a console app in C# they are super easy to understand, same with a hello world app in Node.JS.

They looked perfect for what I wanted to do.  I wanted to read in log files as they were created, parse the content out of them & them log that to an Azure SQL Database.  From there we just attach Excel to the DB and go to town with graphs and pivot tables etc.…

Microsoft.Azure.WebJobs.Host: Invalid container name: $logs.

Problem! Currently it looks like you can’t trigger a Function off blob creation of a log file in the $logs container which is where the files are dropped 🙁  I don’t know if this is something that will change, but ill try and find out.

This means that for now we would have to do one of two things:

1. Manually copy log files to another container periodically. Ok solution.
2. Make another function that is triggered every 15 mins or so to copy the log files out to another container which would trigger or other function. Better solution.

I opted for copying the blobs on a schedule.  I wrote some pretty crude code that:

1. Enumerates all the blobs in the source $logs container
2. Checks if they exist in the destination container
3. Copies them over if they don’t

I say this is crude because it doesn’t keep track of the last blob that it copied and therefore each time it runs it enumerates all the logs each time. This isn’t ideal and I will probably need to make it a bit more sophisticated in the future. But for now running it once a day shouldn’t be a problem.
Update:  Since I published the post I updated the copy code to check to see if it needs to sync files since the last time the function ran + 1 extra day (to be safe).  This means it wont scan all your logs every time.

So here is what we have end to end …

1. Copy logs from $logs in one subscription to /showlogs in another subscription
2. Process each log file as it arrives and put the data in the Database

Show me the code!

Ok so how did I do this.  Below is the two Azure Functions.  I have tried to comment them so you can follow along.

Copying log files from /$logs to another Azure Blob container.

Next, Processing those log files when they arrive.

I have put all the source for these two functions in a GitHub repo here: https://github.com/LoungeFlyZ/AzureBlobLogProcessing

There are two files for each function:

• project.json – contains some dependency information for libraries that I used in each function
• run.csx – the main azure function code

Hopefully someone else will find this useful!

Azure Functions provide a really handy and simple way to run code periodically.  If you are familiar with Node.JS, C#, Python or PHP you should go take a look at them.

-CJ