Working with Bootstrap, Less and Azure

My website www.music4dance.net runs as an application service in Windows Azure and uses Boostrap 3 for styling. Several years ago I explored ways to use Less with Bootstrap in order to set up different themes for different parts of the site and landed on using BundleTransformer to compile the Less files on the fly.  At the time it seemed like the best solution and worked well for several years.

The fragile part of this solution is that it depended on having a JavaScript engine available.  On a machine that I control, this is no big deal.  But in the context of an Azure Application Service I don’t control the image and don’t really know what is available.  I don’t recall exactly how I landed on this solution, but I found that of the options listed in the JavaScriptEngineSwitcher that JavaScriptEngineSwitcher.Msie worked for my purposes.

When my website mysteriously started crashing a couple of days ago, I discovered that this solution was no longer working.  After a bit of instrumenting, I discovered that the line of code that was performing the bunding was throwing an exception:

During translation of LESS code, readed from the file ‘/Content/bootstrap/music-theme.less’, to CSS code error has occurred.

See more details: A fatal exception has occurred in a JavaScript runtime

Check out the full call stack if you’re interested.

My first thought was to try a fully managed version of a JavaScript enginge.  JavaScriptEngineSwitcher supports two – Jint and Jurassic.  Jurassic didn’t work even on my local machine, it threw a stack overflow exception.  Jint actually worked on my local machine but was incredibly slow.  When I attempted to deploy to Azure it just appeared to hang.

Now, I’m reasonably sure that the Chakra engine and possibly even V8 is running on an Azure application service.  But since I couldn’t figure out how to verify that or which version they were running I decided to look for a different solution.  Since I only have four themes there is no real need to generate them dynamically, I can just build them at compile time and deploy all four .css files.

So that’s what I’m doing.  Mads Kristensen’s web compiler does a perfectly good job of compiling Less to Css on save and at build time.   And since I’m using git for source control these days, I can just let it add the generated .css files to my project and deploy automatically (in older SCC systems I would have had to worry about the files getting locked).  No more dependencies on server-side JavaScript engines and there’s a small bonus that I suspect that the site spins up a bit faster.

I would still be interested in knowing what JavaScript engines are available on Azure Application Service instances and how I can determine versions so that I could test on a local instance, but it is no longer urgent since the web compiler solution is a more than adequate substitution.

Call Stack for BundleConfig Exception:

Void InnerTranslate(BundleTransformer.Core.Assets.IAsset, BundleTransformer.Less.Internal.LessCompiler, Boolean)
at BundleTransformer.Less.Translators.LessTranslator.InnerTranslate(IAsset asset, LessCompiler lessCompiler, Boolean enableNativeMinification)
at BundleTransformer.Less.Translators.LessTranslator.Translate(IList`1 assets)
at BundleTransformer.Core.Transformers.TransformerBase.Translate(IList`1 assets, Boolean isDebugMode)
at BundleTransformer.Core.Transformers.TransformerBase.Transform(IList`1 assets, BundleContext bundleContext, BundleResponse bundleResponse, VirtualPathProvider virtualPathProvider, Boolean isDebugMode)
at BundleTransformer.Core.Transformers.TransformerBase.Process(BundleContext bundleContext, BundleResponse bundleResponse, Boolean isDebugMode)
at BundleTransformer.Core.Transformers.TransformerBase.Process(BundleContext bundleContext, BundleResponse bundleResponse)
at System.Web.Optimization.Bundle.ApplyTransforms(BundleContext context, String bundleContent, IEnumerable`1 bundleFiles)
at System.Web.Optimization.Bundle.GenerateBundleResponse(BundleContext context)
at System.Web.Optimization.Bundle.GetBundleResponse(BundleContext context)
at System.Web.Optimization.BundleResolver.GetBundleContents(String virtualPath)
at System.Web.Optimization.AssetManager.EliminateDuplicatesAndResolveUrls(IEnumerable`1 refs)
at System.Web.Optimization.AssetManager.DeterminePathsToRender(IEnumerable`1 assets)
at System.Web.Optimization.AssetManager.RenderExplicit(String tagFormat, String[] paths)
at System.Web.Optimization.Styles.RenderFormat(String tagFormat, String[] paths)
at System.Web.Optimization.Styles.Render(String[] paths)
at ASP._Page_Views_Shared__Layout_cshtml.Execute() in D:\home\site\wwwroot\Views\Shared\_Layout.cshtml:line 42
at System.Web.WebPages.WebPageBase.ExecutePageHierarchy()
at System.Web.Mvc.WebViewPage.ExecutePageHierarchy()
at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)
at System.Web.WebPages.WebPageBase.<>c__DisplayClass3.<RenderPageCore>b__2(TextWriter writer)
at System.Web.WebPages.HelperResult.WriteTo(TextWriter writer)
at System.Web.WebPages.WebPageBase.Write(HelperResult result)
at System.Web.WebPages.WebPageBase.RenderSurrounding(String partialViewName, Action`1 body)
at System.Web.WebPages.WebPageBase.PopContext()
at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)
at System.Web.Mvc.RazorView.RenderView(ViewContext viewContext, TextWriter writer, Object instance)
at System.Web.Mvc.BuildManagerCompiledView.Render(ViewContext viewContext, TextWriter writer)
at System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResult(ControllerContext controllerContext, ActionResult actionResult)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext controllerContext, IList`1 filters, ActionResult actionResult)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<>c__DisplayClass2b.<BeginInvokeAction>b__1c()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult asyncResult, ExecuteCoreState innerState)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.<BeginExecute>b__15(IAsyncResult asyncResult, Controller controller)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.EndExecute(IAsyncResult asyncResult)
at System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)
at System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)<span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_start"></span>

How to Remove Client Secrets from Source Code: Part II (the environment part)

Or:  How to batch update environment variables in Azure App Services.

It’s so easy, yet so wrong to put client secrets and passwords to external services in your source code.  But for a small site like music4dance, it just didn’t seem to be worth the trouble to do anything else when the only people that had access to the source code were also trusted enough to have access to the passwords.  I talked about why I changed my mind in my last post as well as how I refactored my code to make it reasonably easy to make the changes.

The other part of the problem is that I now have a couple of dozen environment variables that I want to set on a number of machines, both locally and in the cloud and I don’t want to have to update them manually.  I can easily write a command or PowerShell script to handle local machines, but how do I manage the Azure App Services in the cloud?  It seems like PowerShell remote management would be a reasonable way to approach this, similar to the solution to restarting Azure Search, but I was completely stumped on how to to do that.

Azure Resource Explorer

Until I found this new tool called Azure Resource Explorer.  Check out David Ebbo’s video describing it here:

Azure Resource Explorer

It’s incredibly straightforward.  Just open up the website https://resources.azure.com and sign in with the same credentials that you use for the Azure portal.  There is a drop-down control in the top center where you choose your directory, make sure you set it to the directory that you’re interested in modifying if you have multiple subscriptions set up.  Then take a gander at the tree control in the left pane.

I see two top-level nodes: Providers and Subscriptions. Spend some time browsing through this tree as there is plenty of gold here.

In order to change the environment variables for my web site, I’m looking for the settings for the App Service called music4dance which is housed in the Default-Web-WestUS resource group.  So I navigate through the hierarchy like this:  subscriptions->bizspark->resourceGroups->Default-Web-WestUS->providers->Microsoft.Web->sites->m4d->config->appsettings.

Once I’ve selected app settings in the tree, I can look at the main panel, where the default tab is labeled Data (GET, PUT).  The text box has a JSON formatted object that describes the node in the tree and contains a property set of the environment variables for app settings.  We can almost stop here if we want to.  The first time through this I just copied my block of environment variables into the properties section and hit the POST button.  I went back to the azure portal, navigated to the Application Settings panel and scrolled down to the app settings panel, and there I saw my new environment variables.

Azure Resource Explorer
Azure Resource Explorer

PowerShell

But the best part is the PowerShell tab. This gives the template to view and modify the object that we’re viewing.

# PowerShell equivalent script

# List appsettings
$resource = Invoke-AzureRmResourceAction -ResourceGroupName My-Resource-Group-Name -ResourceType Microsoft.Web/sites/config -ResourceName my-resource-name/appsettings -Action list -ApiVersion 2015-08-01 -Force
$resource.Properties

# SET list
$PropertiesObject = @{
	#Property = value;
}
New-AzureRmResource -PropertyObject $PropertiesObject -ResourceGroupName My-Resource-Group-Name -ResourceType Microsoft.Web/sites/config -ResourceName my-resource-name/appsettings -ApiVersion 2015-08-01 -Force

You can just copy this into a .ps file and add in your environment variables to the PropertyObject. Don’t forget to add all the other environment variables (you can grab these in an appropriate format from the get/post tab of the resource explorer). To match up with the code that I wrote to load the environment variables, my version of SET list looks something like this:

# SET list
$PropertiesObject = @{
    #Property = value;
    "xbox-client-id" = "my-xbox-clientid";
    "xbox-client-secret" = "my-xbox-client-secret";
    "spot-client-id", "my-spotify-clientid";
    "spot-client-secret", "my-spotify-client-secret";
    # and many more
}

Putting it all together

The way I run this script is to load it into the PowerShell ISE, log into the correct subscription (see the instruction to do that here) and run the script. Pretty simple, and not prone to typos or copy/paste errors the way doing the same thing in the azure portal would be.

I think the ultimate solution would be to write a pair of PowerShell scripts, one to do the local load and one to do the remote load, both taking the same tab separated or JSON formatted file of id/secret pairs. That’s more effort than I would put in for myself, but if others are interested I’ll see what I can throw together.

How to Remove Client Secrets from Source Code: Part I (the code part)

I wanted to share the music4dance code with a potential employer, but I didn’t want to share all of the various keys and secrets that I use to shared login services, music search services and such.  Since I hadn’t previously shared the code with anyone other than people that I would trust to share the keys as well, I have been pretty sloppy with just checking them in with source code.  This is particularly nasty since that meant that if I actually wanted to share the git repo, I would not only have to clean up the code, but I’d have to change every one of the client secrets and passwords.

It’s pretty well documented in a number of places that a reasonable way to do this is to save such sensitive information in the environment and load it at runtime.  That way it will never get checked into the source code.  The problem that I haven’t seen a nice solution to online is dealing with a whole bunch of these at the same time.  There are two parts to this, the changing of the source code and setting up the environment on all the machines that need it.

The documented way to load an environment variable is straightforward:

public string MyKey => Environment.GetEnvironmentVariable("mykey");

And if there’s some possibility that the property is read multiple times (and generally there is), you can easily cache the variable (I love some of these new C# syntax enhancements).

public string MyKey => _myKey ?? 
    (_myKey = Environment.GetEnvironmentVariable("mykey"));
Private string _myKey;

Now, I’m doing this in a dozen or so places in my code, and almost all of them conform to a pattern where I am storing a pair of values.  Either client key and secret or username and password.  So I threw together a little class to handle this.

public abstract class CoreAuthentication
{
    protected abstract string Client { get; }
    public string ClientId => _clientId ?? 
        (_clientId = Environment.GetEnvironmentVariable(Client + "-client-id"));
    public string ClientSecret => _clientSecret ?? 
        (_clientSecret = Environment.GetEnvironmentVariable(Client + "-client-secret"));

    private string _clientId;
    private string _clientSecret;
}

The idea for this is that I had a number of specific authorization handlers for each of the music services that would extend this class and override the Client property to let CoreAuthentication know what environment variable to read.

Then for the simpler cases where I just needed to grab the info and use it to an OWIN security provider, I created a generic subclass that just takes the client name that I use as the base of the environment variable name.

public class EnvAuthentication : CoreAuthentication
{
    public EnvAuthentication(string client)
    {
        Client = client;
    }

    protected override string Client { get; }
}

I use the EnvAuthentication class in my startup authorization like so:

var fbenv = new EnvAuthentication("fb");
var fb = new FacebookAuthenticationOptions
{
    AppId = fbenv.ClientId, 
    AppSecret = fbenv.ClientSecret
};

Not rocket science, but a somewhat cleaner solution than having Environment.GetEnvironmentVariable calls all over my code.

I’ll attack the problem of loading up all of those environment variables next time. Especially the case of batch loading into the azure app service, which I found particularly vexing.