At the Microsoft Build conference this year the Azure AD team announced an interesting new capability called Publisher verification. The goal of this is to let you know, as someone about to give an application permissions to some or all of your data, that the application is from who it says it’s from.

Consenting to give an application permissions to your data is an important acknowledgement that you trust that application with your data. You want to be really sure you are saying “yes!” to the right company.

So … What does this look like?

Publisher verification means that when you are consenting to an application you will see a blue check/tick stamp next to the publisher name like this:

This means that when consenting to an application you can be sure the app you are consenting to is from the publisher that you expect. It’s an extra level of confidence that you didn’t get phished into consenting to a rouge app that is about to steal all your data and do nefarious things with it … well … you trust that the publisher won’t since you trust them right? 🙂  More on this in a moment.

How does this work?

Microsoft is attempting to help customers consent to app publishers they trust by making the app publishers jump through a few hoops to prove they are who they say they are. Most app stores do this like Apple by doing a business verification check for example.

App developers must associate their application registration to their Microsoft Partner Network (MPN) account. The app developer pops their MPN id into the app registration and MS then verifies the following:

1. The verified publisher domain of the app registration in the Azure AD matches a verified domain in your MPN account.
2. The account you are logged in as is an authorized user in the MPN account

The publisher domain requirement, #1 above, could be a little tricky for some developers to meet. You will likely need to find whoever manages your MPN account and work with them on ensuring you have a DNS verified domain verified in both MPN and your Azure AD where you register the app.  A lot of developers register apps in a secondary Azure AD tenant away from their companies primary tenant so this could take a bit of planning to connect up a verified domain in AAD and MPN to make it work.

How do you set it up?

You set this up in Azure AD on the app registration blade for your application.  Go into the Branding tab for your app and you should see the Publisher Verification section towards the bottom. Simply drop in you MPN id.  If all the checks pan out (see above) then your application will be marked as verified like so:

Where is all this going?

As Microsoft holds more and more of organizations data it’s incredibly important they help customers protect it. Ensuring customers are aware of who they are giving access to their data is an important first step in that. That said, it’s certainly not going to stop an app developer from taking all your data if they break that trust with you, but it can help with ensuring you are saying yes to the right app to start with.

In addition to the announcement about publisher verification, Microsoft said they will allow customers to enforce policies that ensure users can only consent to apps from verified publishers.  This would lower the likelihood of a user being tricked into consenting to an app that a malicious actor set up to siphon data etc… They would need to have jumped though more hoops at least to verify their app first.

I believe we will see this taken further in the future in a few ways:

1. Publisher verified app consent only on by default. (see update note below)
2. The ability for admins to set policy that allows consent only to a set list of verified publishers

Microsoft holds the keys to many organizations valuable data.  They really want to avoid a Cambridge Analytica situation and are taking the first steps necessary to build more trust into their APIs and apps that connect to your data.

I think this is a great move by Microsoft.  It might seem like a small one currently, but hopefully we will see more advances in the future that will help us all protect our data.

You can read more about how to set this up with your applications here: Publisher verification (preview)

UPDATE: Publisher Verification is now GA.  Also you must have publisher verification done for your new multi-tenant apps you register after Nov 8th 2020 or users will not be able to consent to them. 

-CJ

KubeCon 2018 was in Seattle this week and I attended as media for the Microsoft Cloud Show. KubeCon is a conference dedicated to the Kubernetes, it’s community and it’s thriving ecosystem of partners and vendors. This year saw 8,000 people attending so it definitely isn’t some small time event.

 

I was really interested in this event for a few reasons, not least of which was seeing what Microsoft was up to there. Like other events my primary interest was walking the expo floor and seeing what vendors and partners were there and what they were doing. It’s a great way to measure the pulse of an ecosystem. You can see who is investing in an ecosystem, get a feel for the excitement, see who is no longer participating and talk to people to get a sense of what is new and interesting.

Microsoft have been investing in Kubernetes for a while now and their presence at this conference was no exception. It was actually pretty incredible to see their booth buzzing at the show with plenty of people at it asking questions and seeing what Azure was all about. I commented on twitter that it almost brought a tear to my eye seeing the Microsoft booth that busy at a confernce about a technology that could certainly historically be classified “non-microsoft” developer friendly. Seeing Microsoft stand alongside AWS and Google Cloud with a solid offering and being taken seriously was awesome!

Microsoft have been doing really innovative things in the containers space recently, for example the public preview of the virtual kubelet. Virtual Kubelet brings the world of serverless together with kubernetes. You can take advantage of Azure Container Instances with Kubernetes so that you don’t need to worry about compute capacity on your worker nodes. Run as many containers as you like and Microsoft will take care of the compute behind the scenes. It also works for other serverless container platforms like AWS Fargate too.

I checked out some interesting vendors in the expo like Atomist who make a platform that helps you build the software delivery process you want. Sysdig that does interesting things around monitoring and securing containers. And Rookout that lets you debug running applications in Kubernetes. It was really interesting to see these vendors providing similar solutions to offerings in the MS dev space that we have grown used to over the years with VS and Azure over the years. It really makes you realize how behind the rest of the industry are when it comes to developer tooling. MS is of course catching up in the Kubernetes space, but .Net tooling is incredible vs. whats available for non-MS developers.

Finally, something i really noticed was how different a conference that was not run by a big vendor. like Microsoft, was. It had a very different feel having the big players as just partners and not running the show. It really rippled across everything from session content not being massaged by the big corporate organizer, the keynotes not being all about one vendor and through to things like child care on site that just seemed so logical.
It was a really great show and I hope I get the opportunity to go again!

When you use a flight tracking app on your phone to see where a flight is it’s very possible that location data has been crowd sourced. Pretty cool!

Sites like FlightAware.com and FlightRadar24.com use feeds of data from people around the world to help build their datasets. Participating in those feeds is open to anyone who has some basic equipment. This works by listening to the ADS-B and Mode-S signals transmitted by aircraft.  These signals identify aircraft and in some cases include positional data. It’s very easy to listen for these signals using a 1090Mhz antenna and an ADS-B receiver. A couple of years ago I bought some equipment on Amazon, hooked up the software running on a Raspberry Pi and got started feeding it to FlightAware.com. 

However, recently I stepped things up a notch with a new better antenna and dockerizing my setup. More on that in a moment, but first …

Getting started – for those who want to try this themselves

If you want to try this out yourself you will need some basic equipment:

• Raspberry Pi
• ADS-B USB stick
• Antenna

You can buy kits with everything you need from FlightAware on Amazon with everything included.

Once you have your equipment the best place to start is with PiAware – FlightAware’s Raspberry Pi pre-configured software. It walks you through everything needed to get you up and running and feeding their network with your juicy tracking data.

You should  be up and feeding the network in an hour or two:

The “good” with the PiAware guide is it’s the simplest build process, the “bad” is that it’s specific to flight aware and doesn’t set you up to feed other providers.

“Need more input!” – Short Circuit (1986)

Eventually you might find yourself wanting more “range”.  The small indoor antenna might let you track aircraft 30mi/50kms away, depending on the terrain around you, line of sight and trees etc…

When you “need more input” you will need a better antenna. This may possibly require some WAF (“wife acceptance factor”) (or HAF, husband acceptance factor) … as it will likely require putting something outdoors and, for best results, on your roof.

I recently upgraded to the FlightAware made outdoor antenna.

My situation called for mounting it externally on the roof along a gutter line.  Ideally I would have mounted it on a peak of the roof, but I didn’t feel comfortable drilling holes in my roof, so opted for a mount that let me hang it from under the eaves.

Satellite Under Eave Mount 1 5/8

Here is what the setup looks like mounted.

The new outdoor setup and better antenna really bumped up my coverage.  Even without optimal mounting (as you can see there are trees on the south side of our house) range went from < 50mi to ~150-200mi in some directions.

Dockerizing all the things

I like Docker containers. They make my life simple for running different apps and services on one box and it seemed to make sense to me that you should be able to run the piaware software and dump1090 software in containers instead of on the Raspberry Pi directly.

I came across an article “Get eyes in the sky with your Raspberry Pi” by Alex Ellis who had done just that! In Alex’s setup however the configuration of the containers is baked into the docker images at build time which isn’t ideal.  I made some improvements like moving all configuration to Environment variables and added Docker-Compose support.

You can find the code and instructions here: https://github.com/LoungeFlyZ/eyes-in-the-sky

With everything in Docker containers it was relatively simple to add a feeder to another tracking site FlightRadar24.com.  They also provide software “fr24feed” that takes a feed from dump1090 and processes/uploads it. You can find optional instructions in the ReadMe file on how to add this pretty simply.

Summary

I love this stuff.  It’s a fun project with hardware and software aspects to it.  Hanging out of a second story window being held by my wife around the waist was a “hilarious” exercise that I suggest every marriage attempts at some point.

I still have some re-wiring to do in the attic to secure the wiring a bit more, and possibly add some more feeders to feed other tracking sites before I’m complete with the project too. 

Going forward I’m not sure what is next for this project yet.  I’m sure there is more to be done and that I’ll likely be mounting more hardware the roof at some point! LOL.

I hope you can enjoy the frivolity of a project like this as much as I do!

-CJ

Call me weird … but i like clear ice in my drinks for some odd reason. So I went on a mission to learn how to make it at home.

Note: This is amost totally pointless. It tastes the same, is just as cold (i think)… but it looks sweet!

Here is how I do it:

-= Making the Ice =-
1. Boil lots of water
2. Let it cool to room temperature (dont skip this step)
3. Boil it again
4. Let it cool to room temperature (yes, again)
5. Fill a small cooler with the water. Leave about 10cm (4″) of room at the top
6. Take the lid off the cooler or leave the lid open
7. Put the cooler in your freezer
8. Leave it for 24 hours
9. Take the cooler out of the freezer and it will be frozen on the top 8cm (3″) or so
10. Slide a knife down the side of the block and ease it around the block. This will let in air and release the block
11. Take the block out carefully and use a knife to remove sharp bits

-= Cutting it/Shaping it=-
Note: I have not mastered this bit yet

1. get a saw or knife and score the block where you want to cut it.
2. Place knife on score line and knock/hit it with a rubber mallet
3. Hopefully it will break cleanly
4. repeat until you have the rough blocks you want
5. Use a hot fry pan to shape the blocks and make perfect sides of the block

Bask in the glory of your perfectly clear ice.

-CJ

Today Microsoft announced the much anticipated competitor to Slack called Microsoft Teams.

At the core Teams is all about lightweight team to collaboration spaces that include in persistent chat, docs, notes, instant message, video calls and more. It really brings together many parts of Office 365 into one experience and is simple and easy for users.

 

Why is this so important for Office 365? Well, in short, Microsoft has had all these pieces of technology for many years but has not brought them together in a cohesive application experience for users …. Until now.

I’m excited about Microsoft Teams! For some time many have marveled at Slacks simplicity and therefore friction free collaborative experience, but winced at having to manage, purchase and integrate yet another application. Teams is baked into Office 365 and it comes for free. Is it as good as Slack? We don’t know yet. It’s only in preview currently and only time will tell. But it’s off to a good start.

Reliance on people profile data is more important than ever

Virtually all the new experiences going into Office 365 have people at the center of them. Microsoft Teams is no different.

People are at the center of Microsoft Teams. However, to get the best experience in Teams, and to make it as compelling and interesting as the demos suggested, you must have great people profile information.

Org charts

Take for example the org chart they showed in the demo. Looks great and gives you all the information you need to know when looking where in the organization someone sits. It’s also interactive and you can navigate around etc…

But 75%+ of companies don’t have the org structure data needed in Office 365 to power this feature!

 

Photos everywhere

Or take the photo thumbnails that show everywhere to help you connect with the person you are IM’ing with and make it more personal.

 

Those photos come from Azure AD and Office 365. If you don’t have your users profile photos in AD then chances are you will be getting a subpar experience in Teams.

Bots and automation

WhoBot was another example of how people profile information is powering new experiences. WhoBot is a chat bot that lets you look up people in the organization based on their name or skills for example. Again, all this is driven from people profile information in Office 365.

So what if you don’t have great profile data?

Most organizations don’t have great data in Active Directory, Azure Active Directory or Office 365 profiles. So don’t worry, you are not alone 🙂

This is why we created Hyperfish to help organizations understand their profile data and then complete or fix the missing or incorrect information.

Our mission is to make experiences like Microsoft Teams, Delve, people search experiences everything they should be with complete and accurate profile data.

If you don’t know what how good you profile data is you can try our free analysis app! It support Active Directory on-premises as well as Azure Active Directory / Office 365.

Try it out here: https://app.hyperfish.com

If you want to fix your data you can have Hyperfish reach out to users through email or IM to gather it from the folks missing data. e.g. missing photo photos, incorrectly formatted phone numbers, non-standard addresses, non-standard job titles etc… Learn more here about the full Hyperfish product: https://hyperfish.com/

At Hyperfish we are pumped about Teams! The preview looks great and we cant wait to see what else comes as part of it before it is due for release in Q1 of 2017.

We hope you enjoy these new capabilities and get in touch if you need a hand with keeping your people profile data clean and up to date!

-CJ

At Ignite 2016 I did a quick 20 min theatre session in the expo hall on driving better adoption with your Office 365 rollout by leveraging people centric features. A lot of organizations dont make it part of their plan to ensure they have the right people profile data set up to make the best use of the ever expanding range of experiences in Office 365 that rely on it.

They are letting their users down, IT down and the business down.

Successful adoption means focusing on users

In Office 365 most of the profile data comes from Azure Active Directory. Sadly, it’s all too common to see it poorly populated.

As an example, here are some charts from our Hyperfish Directory Analyzer tool that are very common:

Missing this data impacts your Office 365 deployment. Some are simple yet important and others are business criticial.

Here are the Top 10 things I felt are the most common things people are missing out on due to poor profile data that drive better user adoption and use of these tools.

#1 – Delve People Profiles

Microsoft is investing in making the Delve people profile pages THE definative profile page in Office 365 for users. Search etc… will use these. Without great data these look bland and dont provide users the information they need to find and connect with people quickly.

#2 – Contact cards everywhere

Contact cards pop up in all sorts of places e.g. in Outlook (clicked 10s of millions of times a month no less!) Without great data they make it hard to connect and discover people.

 

 

#3 – New SharePoint experiences

SharePoint is in the middle of a visual makeover in Office 365. More people data drive experiences like the new rich contact panels will be popping up all over the place.

#4 – Mobile SharePoint Intranet

Along with better desktop web experiences, SharePoint have released new mobile apps to help you find and connect with people while you are on the move. Great for finding someone’s phone number while out and about.

#5 – Office apps

There are plenty of places in Office applciations like Word and Excel where people information pops up. For example while co-authoring a document with someone else. Good profile data makes it a much nicer experience for seeing who is in the document.

#6 – Dynamic Distribution Lists in Exchange

Not many people know, but you can create dyanmic distribution lists that include people that have specific words in their profiles. Great way to ensure that all Sales people are in the right DLs!

#7 – Workflows

One of the most common people want to do in workflows is excalate a task to someone’s manager if they out of the office or take too long to respond. Without good organization heirachy data in profile this isnt possible! This means people have to write more code to look it up from somewhere else e.g. HR system, or replicate the data somewhere else (which will become stale the day after).

#8 – People Search

Here is a test. Go to your search portal in Office 365, flip to the people search tab and type “Sales”. Did you get great results? Probably not. People search is driven by people profile data and without it search is hard to use and find the right people at the right time.

#9 – Groups

Groups are slowly becoming the new team sites for projects etc… They are rich with people data and conversations. Make sure they sing with decent profiles.

#10 – Skype for Business

Skype for Business uses people profile data for its Search, dialing and in calls. If you dont have good profile data you cant find the right people in search, you cant dial people if they have poorly formatted phone numbers and you dont get to see who you are talking to without nice profile pictures. Help remote users connect with one another by ensuring their experience rocks.

Summary

These are just a few of the experiences I think most people are missing out on in Office 365 due to poor profile data. I founded Hyperfish to help people whip their people profiles into shape and start making these experiences rock for users thus driving satisfaction, adoption and reuse of the tools you have already bought.

I hope you are getting the most out of your Office 365 investment and not letting something like profile data get in the way of users loving it.

-CJ

Exciting news! Last week we released a beta of our free directory analysis product! 

This is the first small step towards a much larger product we have coming later in September at Microsoft’s Ignite conference for helping organizations leverage their investments in SharePoint and Office 365 better. It will ensure organizations have rock solid people profile data in their directories to power experiences like profiles, Delve, search, personalized intranets, automated business process decision making and more.  But more on that in the coming weeks (stay tuned to www.hyperfish.com also).

So what does this directory analysis do?

Simply put, it scans your on-prem Active Directory or online Azure Active Directory and tells you interesting facts and figures about the completeness and accuracy of your users profile data. We send you a report on what we find.

 

We have been testing it with our beta users and now we are ready to have others try it out.

It’s a Beta, you might hit a bug and hopefully will have feedback for us! If you do, please reply to the email you get from us with your thoughts.

Try it out here:  https://app.hyperfish.com

We would love any feedback you have! Also feel free to reach out if you hit any issues on twitter @LoungeFlyZ

Thanks!

SharePoint and I go way back.  I have a special place in my heart for SharePoint.  I got started with the product back in 2002 and it’s taken me places.  It brought me to the USA when I transferred with Microsoft to join the Windows SharePoint Services engineering team in 2007 and onto the Technical Product Management team in 2009.  I owe it a great deal.  Its for this reason that although my new day job isn’t directly tied to SharePoint any longer that I care about its future.  I lived its glory days in 2009-2012.  SharePoint fever was rampant. SharePoint Conferences sold out at 12,000+ people.  There was a gold rush in the enterprise collaboration space orbiting SharePoint.

I want SharePoint to rise again as a fantastic platform for enterprise collaboration.

I make no bones about the fact that I haven’t been SharePoint’s greatest proponent recently.  I feel that Microsoft was chasing shiny things at the expense of what made SharePoint great to begin with.  Things felt 80% done in it’s Enterprise Content Management and Collaboration capabilities.  Like a theme park that was great for a while but started to age and you couldn’t help but think what could have been if only it had been updated.  The SharePoint team went off and started building other things vs. focusing on updating its core features like Team Sites and document collaboration.  Competitors like Dropbox showed what a better experience could be like as SkyDrive (now OneDrive for Business) fell further behind.

All this flailing about drove me nuts.  I got disheartened honestly at Microsoft rewarding engineering for shipping new things that no one wanted while letting their golden goose slowly die.

There is nothing quite like a leadership change to signal to everyone that the status quo isn’t working.

Over a number of years SharePoint fell under a number of leaders who really didn’t understand the enterprise software in my humble opinion.  They came from the consumer side of Microsoft.  SharePoint got little attention.  (PS: who lets a multibillion dollar business stagnate the wither like that anyway?!!? nuts!)

Last year Jeff Teper returned to lead the SharePoint Product Group.  Many are hailing this as a turning point.  The return of the king in many ways.  After all … he is the father of SharePoint.

Jeff round 2…

Today we saw the first step in Jeff’s new era of SharePoint.  Jeff has put the first stake in the ground for his new vision for SharePoint and where he wants to take it.

…  empower people, teams and organizations to intelligently discover, share and collaborate on content from anywhere and on any device.

It focusses on four key areas:

• File sharing on any device
• Intranet/Team sites on any device
• New extensibility options for developers
• Enterprise features around security, compliance and privacy.

What I LOVE about this that it focusses on core SharePoint capabilities.  It’s radically updating what is already there vs. adding something new.

Someone finally got the memo that to stay relevant you need to focus on what made you great to begin with, don’t lose sight of your fans and incrementally build from a solid base.

SharePoint lost it’s way … the real question is … is all this too late?

I don’t think so.

Many think SharePoint’s game is over. There are a multitude of smaller more nimble productivity apps out there that do a lot of what SharePoint introduced to businesses back in 2001.  But the one thing many forget is Microsoft’s ability to execute end to end when the stars align just right.  Building the features is just the first star to align.  Company vision is another, Marketing is another, Sales is another.  When all these align at Microsoft it is an unstoppable force of nature.  None of SharePoint’s competitors can touch it when these things all come together end to end.  But it hinges on it being a core imperative at the company to make it happen.  I feel there is a very good chance of that given Satya personally pushed asked Jeff Teper to lead the SharePoint group again.  He understands how critical owning File sync and Share is, how crucial owning team collaboration is and how important personal productivity is.

Here’s Satya talking about SharePoint and OneDrive at today’s event

New product experiences…

Like I already mentioned, the first star needed for perfect alignment is product.  It all starts with compelling product capabilities.

Team sites vNext

Today the SharePoint team showed new product capabilities primarily in the team and document collaboration space with fresh web and mobile experiences.  This in my view is playing catchup for the last few years of rot in the SharePoint UX.

Honestly … it looks great and end users who are not used to existing SharePoint sites will love it.  Users already used to SharePoint will likely be confused about where the ribbon interface went and have a hard time finding things in the place they were used to.

This new UI is not particularly innovative or over and above what competitors have already, but it is super important table stakes that were desperately needed.

There is a new SharePoint mobile app.  To be honest, I am pretty ho hum about this.  In my experience companies give their intranet a brand.  Microsoft does with its intranet, MSW (MS Web).  I think SharePoint missed a great opportunity to give companies the ability to white label this app and deploy “MSW” to peoples phones, brand it with their colors and customize it with additional apps and capabilities.

Finding sites…

Almost since the dawn of time one of the issues with SharePoint was the rampant proliferation of SharePoint sites with little to no ability to find any of them easily.  SharePoint are revamping the Sites experience in O365 to show you all the sites you belong to.

Nice improvement!

Flow and Power Apps

This demands a whole post on its own!

SharePoint forms and workflows are still left wondering why they are not getting the attention they deserve.

Flow is a new tool (in preview) to help you build easy lightweight workflows that connect to different systems and tools.  It actually shows quite a bit of promise in my view, but it’s built by the Azure team and you can tell they are used to building things for developers, not end users.  Of course this will likely get better over time.  To really succeed it needs to be deeply rooted in the Office and SharePoint user experiences, a little bit of which is seen in the screenshot below with the “Add Flow” button in SharePoint.

This is one area where either Power Apps + Flow really has to grow up fast, or SharePoint needs to take the reigns of these tools.

Extensibility …

Ahhh … a topic near and dear to my heart.  Prior to leaving MS I ran the technical marketing team in Office 365 that focused on extensibility and the partner product ecosystem around Office 365 and it is still something I deeply care about.

Microsoft has always been a partner lead company.  Steve Jobs famously said the thing he admired most about Microsoft was it’s partner lead attitude.  Extensibility and the ability to build on and beside products is critical to a vibrant and healthy product ecosystem.

SharePoint didn’t get to a $2.5B+ business for Microsoft by being a great out of the box application.  It got there by partner companies building amazing tools, add-ins and products that enhance SharePoint with new capabilities.

SharePoint has had an ongoing identity crisis with extensibility.  Each new version introducing enough of a change to piss off developers.  Although on the one hand these new capabilities are great … developers also hate to have to change development processes and relearn best practices.

They are going to need to do that again this time around.  The fantastic part about it is of course the ability to surface UI directly in SharePoint using modern web development tools and frameworks.

I have some pretty bias views on this.  For a very long time there have been people bending SharePoint UI to do unnatural things.  They inject script and insert, tweak and change things to make the SharePoint UI do what they want.  They were driven to this by customer demands and consultants inability to say no when it comes to $.

This new framework for SharePoint UI gives developers a new UI API.  Documented hooks and points at which extending the UI is condoned and encouraged and something the team will support.  This should be the UI nirvana developers are looking for.  But the proof will be in the pudding and we will see if it goes far enough with its ability to deliver what customers are demanding.  There will be gaps, I’m 100% sure of that … but the key will be SharePoint’s ability to address those, driven in a collaborative manner with the community.

I have some concerns about the new framework with regards to its suitability in building commercial products.  I need to do more digging before I draw conclusions, but from what I know now it is very much targeted at enterprise developers, not ISVs.  The packaging and deployment model for example is different from what we have with Add-Ins today and I feel many of the benefits of the Add-In model that ISVs had, for example end user deployment and code isolation,  are not present in this new framework. I can only hope that a re-focus on these in the not too distant future.

Where to from here…

I feel we are seeing the first glimpse of the stars aligning again in SharePointlandia:

• I am impressed with the teams renewed focus on the core of what are SharePoint’s strengths and that feels good!
• Having this line in the sand is great … they need to do this twice a year and keep the pressure on catching back up to the pack.  They need to do this so that the shiny new things make sense again because the core is strong. You can’t innovate the body of a car if you leave the engine to languish (ala DeLorean).
• I am bullish that Jeff understands this and has rallied the team around this as a core priority.
• I am upset that they have not yet decided this is a core priority for ISVs building products with SharePoint as this is a critical component of what made them great the first time around.

I am also extremely happy to see some focus, care and attention on SharePoint again.

I want SharePoint to rise again.

-CJ

You can hear more about the #FutureofSharePoint in an interview on my podcast the Microsoft Cloud Show here.

 

One of the ways AC and I track how many people are listening to the Microsoft Cloud Show episodes we put out is by using the logs created for Azure Storage.  These track the various requests for the mp3 files for each episode.  You have to turn on this logging for your account and once this is done then log files are written into the /$logs/blob folder of your storage account.  You can read more about Azure Storage Analytics here: https://msdn.microsoft.com/en-us/library/hh343270.aspx

However, the way the storage logs are filed does not exactly make analyzing them easy in something like Excel.   They are logged in /year/month/day/hour/5min chunk folders/files in CSV format. (detail of what is logged is here).

Until recently our process for downloading the analyzing these was rather laborious and included:

1. Download all the logs files.
2. Use some PowerShell to squash all the thousands of CSV files into one big one.
3. Use SQL Import tool to take the CSV and import each line into a temp table in SQL Azure
4. Run a T-SQL script over the data to remove all the rows we were not interested in (we only wanted the logs for downloaded MP3s, not other assets etc.…)
5. Mutate the data in various columns for better reporting. e.g. pulling important parts from the User Agent string and pulling an episode number out of the filenames.
6. Finally inserting all the new rows into the final reporting Table
7. Using Excel to report over the data

It was error prone, painfully slow at times and required all these things to be done each time we wanted new statistics.  At times we would go months without doing this and were flying blind on how many people were listening.  Not really very good.

What we really needed was an automated process!

Azure Function Apps to the rescue!

In March 2016 Microsoft announced Azure Functions.  They are lightweight bits of code that can be run in Azure when certain events occur, like on a timed basis, when an HTTP request is made and when a blob is added to storage.  What is nice about them is that you are not paying for a Virtual Machine to sit there mostly idle.  You only pay for what you use.  You can write them in a variety of languages and trigger them in various ways.  If you know how to write a console app in C# they are super easy to understand, same with a hello world app in Node.JS.

They looked perfect for what I wanted to do.  I wanted to read in log files as they were created, parse the content out of them & them log that to an Azure SQL Database.  From there we just attach Excel to the DB and go to town with graphs and pivot tables etc.…

Microsoft.Azure.WebJobs.Host: Invalid container name: $logs.

Problem! Currently it looks like you can’t trigger a Function off blob creation of a log file in the $logs container which is where the files are dropped 🙁  I don’t know if this is something that will change, but ill try and find out.

This means that for now we would have to do one of two things:

1. Manually copy log files to another container periodically. Ok solution.
2. Make another function that is triggered every 15 mins or so to copy the log files out to another container which would trigger or other function. Better solution.

I opted for copying the blobs on a schedule.  I wrote some pretty crude code that:

1. Enumerates all the blobs in the source $logs container
2. Checks if they exist in the destination container
3. Copies them over if they don’t

I say this is crude because it doesn’t keep track of the last blob that it copied and therefore each time it runs it enumerates all the logs each time. This isn’t ideal and I will probably need to make it a bit more sophisticated in the future. But for now running it once a day shouldn’t be a problem.
Update:  Since I published the post I updated the copy code to check to see if it needs to sync files since the last time the function ran + 1 extra day (to be safe).  This means it wont scan all your logs every time.

So here is what we have end to end …

1. Copy logs from $logs in one subscription to /showlogs in another subscription
2. Process each log file as it arrives and put the data in the Database

Show me the code!

Ok so how did I do this.  Below is the two Azure Functions.  I have tried to comment them so you can follow along.

Copying log files from /$logs to another Azure Blob container.

Next, Processing those log files when they arrive.

I have put all the source for these two functions in a GitHub repo here: https://github.com/LoungeFlyZ/AzureBlobLogProcessing

There are two files for each function:

• project.json – contains some dependency information for libraries that I used in each function
• run.csx – the main azure function code

Hopefully someone else will find this useful!

Azure Functions provide a really handy and simple way to run code periodically.  If you are familiar with Node.JS, C#, Python or PHP you should go take a look at them.

-CJ