2017 on Funky Si's Blog (Dev)

Looking back at 2017

funkysi1701@gmail.com (funkysi1701) — Thu, 28 Dec 2017 20:00:45 +0000

As 2017 starts to draw to a close let’s look at some of the highlights from the past year.

Edward

In November I became a father again and celebrated the arrival of Edward into the world. Before the birth I was worried how well my first son James would adapt to the new arrival but I am happy to say I didn’t need to, he mostly ignores the new arrival.

Star Trek

In September the latest Star Trek TV series Discovery arrived on Netflix. I have been very positive so far about the show, I am still positive however there are things that annoy me about the new show, however I am keen to see the next episode each week so they are definitely doing something right.

Office move

In May I helped my employer move to brand new offices . This was a lot of hard work, but everything went as planned and our new home is great.

Side projects

I blogged in February about having a side project but it wasn’t until later in the year that this solidified into what I am working on now. It started as a way to understand interfaces this then led to learning Xamarin and creating an android app to help promote this blog. Troy Hunt expanded his HIBP API to include passwords that have been in data breaches, I then started working on pwned passwords which is a simple android frontend for this.

Learning R

In Feb I also experimented with a new programming language R . It started with a problem about converting exchange rates and ended with a SQL stored procedure that executed some R code, I still want to learn lots more but it was a good example of the sorts of thing you can build with R.

DNS programmatically

One of my most successful blogs was about how you can create dns records within your application eg sites that have yourname.domain in the url. It was a great chance to look at how dns works and learn more about what azure can do to manage this. I want to expand on this and build something.

So that was 2017, for some ideas about what 2018 might have in store for me look out for my next blog post.

Getting started with Azure Tables

funkysi1701@gmail.com (funkysi1701) — Sun, 17 Dec 2017 20:00:45 +0000

Azure Table storage is cheap way to store data, however it has some drawbacks that you should be aware of.

Azure Table storage is a simple way to store NoSQL data with key/attribute pairs. I am very familiar with storing data in SQL databases and would still choose SQL over Table storage, however Table storage is significantly cheaper so could be worth investigating depending on your project.

Troy Hunt makes use of Table storage for his Have I been pwned? website so there are projects out there that make use of it to great affect.

To work with Table storage you need to use a nuget package WindowsAzure.Storage

Install-Package WindowsAzure.Storage

To load data from a Table in Azure Table storage I use the following code

CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("tablename");
TableOperation retrieveOperation = TableOperation.Retrieve<Entity>(PartitionKey, Rowkey);
table.CreateIfNotExists();
TableResult retrievedResult = table.Execute(retrieveOperation);

Example eg = new Example();
if (retrievedResult.Result != null)
{
 eg.ID = ((Entity)retrievedResult.Result).Id;
 eg.Date = ((Entity)retrievedResult.Result).Date;
}
return eg;

You need to create a class (called Entity in the example above) derived from TableEntity which defines the Partitionkey and Rowkey, plus and other columns you want to store in table storage. The row key and partition key uniquely identify the data in the table, think of this as the primary key of the table if you are used to SQL. This class must also contain a parameterless constructor.

This is the only way to retrieve data, using the partitionkey and rowkey. If you want to retrieve a specific piece of data you would need to retrieve all rows and then search them for what you need. For me this is not a big problem as I only have 150 rows but if you have millions of rows you may need to think carefully how to use this.

To save data I use a very similar piece of code

CloudStorageAccount storageAccount = CloudStorageAccount.Parse(path);
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("tablename");
Entity post = new Entity(ID, DateTime.Now);
TableOperation insertOperation = TableOperation.InsertOrReplace(post);
table.CreateIfNotExists();
table.Execute(insertOperation);

I create a Entity object and then pass this as a parameter into the InsertOrReplace method.

To delete data it is also very similar, you create an entity object and pass this as a parameter to the Delete method.

When debugging my table storage code I found the Azure Storage Explorer very useful for seeing what data actually existed in the table and what might be throwing an error, usually something wrong with my Entity.

I mentioned earlier that Table Storage was cheaper than SQL Azure. Well for my simple playing about with things I have found my monthly charge of £10+ has been reduced to £1+ If I were to build anything that is more than just me learning about how it works I would probably continue to use SQL but for the cost of learning new tech it is well worth giving table storage a try.

Zero code changes outside of git

funkysi1701@gmail.com (funkysi1701) — Mon, 13 Nov 2017 20:00:45 +0000

Source control is bread and butter for web developers, however not so much for SQL developers and other business people. One of my goals for the coming year is to get the whole of my team using source control processes.

One of the arguments against using source control is there will always be a few exceptions where it won’t be used. Lets look at a few scenarios to help make the case for source control.

Scenario One

Web Dev 1 makes a change in production to fix an issue and doesn’t use source control. Web Dev 2 is working on a new feature and makes use of source control. He is given the sign-off to deploy his new feature and in the process undoes Web Dev 1’s fix.

Management blames Web Dev 2 as they “broke” stuff during the deployment. Web Dev 2 doesn’t understand what happened and Web Dev 1 is oblivious to the entire scenario despite in reality being part of the problem.

Scenario Two

A SQL Dev works all night to fix a problem. He deploy to production and doesn’t notice a missed where clause due to lack of sleep.

—

Let’s look at these two scenarios done with the whole team buying into source control.

Scenario One

Web Dev 1 makes a change in production to fix an issue and also commits to source control. This takes him an extra 5 minutes (if that!) Web Dev 2 is working on a new feature and makes use of source control. He is given the sign-off to deploy his new feature and in the process merges Web Dev 1’s fix before he deploys to production.

Management is happy with Web Dev 1, nothing is broken and new functionality has made the website better not worse.

Scenario Two

A SQL Dev works all night to fix a problem. He deploy to production and doesn’t notice a missed where clause due to lack of sleep, he also commits to source control this takes him an extra 5 minutes (if that!).

In the morning a colleague looks through the commits made during the night and queries the missing where clause. A fix is made in source control and deployed, management and the client are unaware that a problem was created and fixed.

—

These are simple scenarios but I cannot think of any situation where the outcome would be better to not use source control. These examples assume no continuous deployment, adding this to the system before every team has bought into source control would cause bugs and issues all over the place.

However with a team that is 100% behind source control continuous deployment can achieve an amazing productivity boost. After every commit code could be automatically tested, reviewed by other members of the team and deployed to test environments for further analysis. Only tested code that has been reviewed can get anywhere near production environments.

Having said all this there are ways especially on the SQL side to mitigate loosing changes while you work on training individuals and convincing management. Before any code is deployed to production do a schema compare. If you only see changes you have made you can proceed, if you don’t shout at your team.

Advantages

Code Reviews
Team awareness of what is being changed
History of changes
Smoother Deployments
Automation
Better Teamwork
Diff changes between different versions
Many more

Disadvantages

Time to learn tools
Remember to commit changes
Technologies that use binary files harder to version control

SQL with Visual Studio Code

funkysi1701@gmail.com (funkysi1701) — Mon, 06 Nov 2017 20:00:45 +0000

Writing SQL queries is typically done with SQL Management Studio (SSMS). However this tool is a bit of a beast so let’s look at how you could use Visual Studio Code instead.

Visual Studio Code is a free text editor but it is so much more than just a text editor. VS Code can be downloaded from https://code.visualstudio.com/Download

To work with SQL Server download the mssql extension. Press CTRL+SHIFT+P and then Select Install Extension and type mssql.

Intellisense in Visual Studio Code is brilliant, better than SSMS. Lets look at how to get it all set up.

Create a new file and set the language type to SQL (Press CTRL+K,M )

Open the command palette, CTRL+SHIFT+P and type SQL to show the mssql commands. Select the Connect command.

Then select Create Connection Profile , this creates a profile to connect with your SQL Server. Follow the prompts to get it all setup.

Look in the bottom right corner of the status bar and you should see you are connected.

Now if you type sql you will see a long list of SQL code snippets that you could use.

Choose a snippet to create, and edit it as required. When you are happy press **CTRL+SHIFT+E ** to execute.

This is basically all there is to it. However this is an incredibly powerful way of working, the intellisense instantly tells you what database objects you can use in your query and there is a wealth of different snippets you can use.

When returning data you get a similar view to SSMS but you can save as Excel, CSV or JSON.

SSMS is a very graphical way of doing things, you can double click a table and see its columns or indexes. VS Code relies on TSQL commands but you have access to exactly the same information.

For more information about VS Code and the mssql extension check out https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-develop-use-vscode

Interfaces are cool!

funkysi1701@gmail.com (funkysi1701) — Tue, 31 Oct 2017 20:00:45 +0000

A while back I blogged about learning about interfaces as I didn’t really understand the value of them. I do now.

I created an application that used interfaces so I could learn how it worked. I created a Logger Interface and created multiple classes that implemented that interface so I could swap out the different implementations easily. I created a SQL Logger and a File Logger and my code could be written and be completely unaware of which implementation it was using.

This application uses SQL Azure and so I have a monthly bill to pay. Wouldn’t it be cool if I could reduce this bill? How about using the cheaper table storage instead?

Easy!

Create a new class that implements my interface and all I need to do is write the three methods defined in my interface and I can swap from SQL Azure to table storage.

Another benefit to interfaces is testing. Say I have an interface called inotification for sending notifications, I can have several implementations of this email, twitter, slack etc

None of these implementations should be used in unit tests, as you don’t want a tweet being sent every time you run your tests. Why not create an implementation that simply returns something for each method call and doesn’t actually do anything. I can then run my tests with my fake implementation which tests my code logic but not the implementation I have chosen (this can be tested later on with integration tests or user testing if required).

This is pretty much all I have to say about interfaces. I just like how I can swap different implementations.

It does take a bit of work to get the interface setup. I found that when writing the second implementation the interface would need to change slightly, mostly as it was badly designed to begin with. I think for beginners there may be some value to writing multiple implementations of an interface so you can be sure your interface is good, however I am sure with experience this will not be required.

Create DNS records programmatically

funkysi1701@gmail.com (funkysi1701) — Mon, 16 Oct 2017 20:00:45 +0000

DNS is the backbone of the internet. It converts domain names into IP addresses. But how can we do clever things with it?

Making DNS changes has always been a pain. You have to remember what company you registered your domain name with, then locate the login details for their website, login and fill in some web form and wait for the changes to replicate across the internet.

There are various websites that assign you a custom subdomain name like name.domain.com, obviously they must do this programmatically, they don’t have a guy following the above process.

Lets see how we could do this.

My first thought was to make use of the DNSimple API. https://developer.dnsimple.com/v2/ The API allows all sort of DNS changes to be made, however this is a paid service so let’s look at other options.

Azure now has a DNS section. You can go to the portal, add a DNS zone, and then add as many A, CNAME and MX records as you need. This is of course no better than doing it manually.

Azure offers a SDK and an example Visual Studio project. Let’s look at how it works.

First you need to setup some credentials to use your Azure subscription. The easiest way to do this is with powershell.

Login–AzureRmAccount
$sp = New-AzureRmADServicePrincipal –DisplayName exampleapp –Password "{provide-password}"
Sleep 20
New-AzureRmRoleAssignment –RoleDefinitionName Contributor –ServicePrincipalName $sp.ApplicationId

This will create a AD Service Principal which has access to your Azure subscription.

You need to get the following IDs from the Azure portal.

subscriptionId for your subscription
tenantId or the Azure AD Directory ID
applicationId of the service principal created above
password you entered into the powershell script above

var serviceCreds = await ApplicationTokenProvider.LoginSilentAsync(tenantId, clientId, secret);
var dnsClient = new DnsManagementClient(serviceCreds);
dnsClient.SubscriptionId = subscriptionId;

var recordSetParams = new RecordSet();
recordSetParams.TTL = 3600;
recordSetParams.ARecords = new List<ARecord>();
recordSetParams.ARecords.Add(new ARecord("1.2.3.4"));
var recordSet = await dnsClient.RecordSets.CreateOrUpdateAsync(resourceGroupName, "funkysi1701.com", "test", RecordType.A, recordSetParams);

The above code connects to Azure and creates an A record for test.funkysi1701.com that points at the ip address 1.2.3.4.

Other DNS records can be created in a similar way.

The above example creates a new RecordSet but you can use the following to delete or get existing records.

var recordSet = dnsClient.RecordSets.Get(resourceGroupName, "funkysi1701.com", "test", RecordType.A);

dnsClient.RecordSets.Delete(resourceGroupName, "funkysi1701.com", "test", RecordType.A);

Now I have found out that I can write a console app to edit my DNS records I need to change the nameservers for all my domains so I can take advantage of this new feature.

Discovery – My thoughts so far

funkysi1701@gmail.com (funkysi1701) — Tue, 10 Oct 2017 20:00:45 +0000

I tried to resist but I am going to have to write about the new Star Trek series Discovery. Warning this post is going to include SPOILERS. If you read on you have been warned.

I watched the first two episodes of the brand new Star Trek TV show. As I am an international viewer I used Netflix, if you are from the US you need to use CBS All Access.

I have heard a lot of complaining that the show is not free to watch. CBS is making Star Trek Discovery to make money, if they don’t make money they will stop making it. Its as simple as that.

Netflix costs £5.99 per month and CBS All Access costs $5.99 per month. I don’t believe this is a lot of money. The 5.99 above allows you to watch any of the 726 Star Trek episodes at any time of day whenever you want as many times as you want, plus you have access to all the other movies or TV shows available. I don’t think this is too much to ask.

Back to Discovery. I loved it! It felt like Star Trek. I had feared it might only share the name and would be an action filled TV show that had little in common with what he had seen before. I was wrong this is definitely a show that can proudly call itself Star Trek.

The Klingons

When it was first announced that the Klingons would feature in the new show I was a bit “meh”. We had done lots with the Klingons before and they had never been my favourite alien race.

The look of the Klingons was also going to be changed, I have to admit this didn’t bother me. Klingons have had their look updated before. In 1966 Klingons had dark faces and smooth foreheads, in 1979 the forehead ridges were added as the first Star Trek movie was made, and now Discovery has removed all hair from Klingons. I can explain it away as productional changes, I don’t need an on screen explanation like we had on Enterprise (or even DS9).

What I have seen so far is a menacing alien race that fits with what we have seen before with lots of references to honor, Kahless and speaking Klingon. I must admit reading all the subtitles is getting a bit tiresome, but that is a minor issue.

Conflict

Historically Star Trek has not featured conflict between the main starfleet characters due to the idea that humanity has evolved beyond this. DS9 got round this by having Odo and Kira who are not starfleet characters so can have a little bit of conflict. Discovery has completely abandoned this idea.

In the pilot episode, the Michael Burnham commits mutiny on her captain, even attacking her with a vulcan nerve pinch. In the third episode we finally meet the discovery crew. Gone are the TNG days where the ships crew are like a family, I am not sure I can think of a single character that would call another character “friend”. We are in a time of war so this would be expected, however I do hope we see deepening friendships form between characters.

I am OK with the change to feature more conflict. I must admit the darkness of the third episode, did stop to make me think a bit, however by the fourth episode I was won over.

Cast

The story is concentrated around Michael Burnham, so I do worry that other characters won’t get a look in. However from what I have seen Michael Burnham is a great character. She is a strong female character, with a intriguing back story relating to Sarek and possibly also Spock.

Saru the alien character on the show, played by the very tall Doug Jones is great. From the trailers he was all about sensing death, but there is far more to him than that. I am looking forward to learn more about him and his threat ganglia.

As the show has gone on we have started to learn about the other characters like Captain Lorca, Tilly and Paul Stamets.

Title Sequence

I don’t like the opening title sequence, it feels very cheap like a draft version which hasn’t been finished yet. I was looking forward to a title sequence that would show off the USS Discovery, maybe like TNG where it warps around our own solar system. What we see is some of the tech that features in Star Trek which is nice but I want more. The theme tune while great that it features parts from the classic theme doesn’t stand on its own. I can hum all the other TV shows themes, and now after 4 episodes would struggle to do that.

However I expect that after a while I will grow to like this more.

Overall

I like Discovery and I will keep watching it. I want to find out what happens to the characters. There are no annoying characters like other shows have had. There is character development, none of the characters are going to finish where they started. I want more friendship between characters, but I expect that will come, I want more exploring and doing Star Trek stuff, but most of all I want more episodes, roll on next weeks show.

Other things I noticed

Sound effects on the bridge are awesome, it makes me feel at home
The USS Discovery doesn’t feature in the first two episodes despite being the title of the show.
Others have noted that Captain Georgiou has books on her shelf which feature classic episode titles

Did you know you can track episodes from Star Trek Discovery in Episode Atlas .

Star Trek is back with Discovery

funkysi1701@gmail.com (funkysi1701) — Fri, 22 Sep 2017 20:00:45 +0000

In November 2015 it was announced that a new Star Trek series was going to be launched. It has been a long wait with multiple delays but Star Trek Discovery is finally here.

Star Trek Discovery launches in the US on 24th September and in the UK on 25th September.

Lets have a look back how we got here.

Star Trek was created by Gene Roddenberry in the 1960s. He imagined a show where we would explore the galaxy while also exploring the human condition. He described it as a wagon train to the stars. However it was cancelled after 79 episodes and three seasons due to budget cuts and low viewers.

After cancellation its popularity boomed in reruns. In the mid 1970s an animated series was created. But demand for Star Trek continued until a new Star Trek show was commissioned called Phase Two.

In 1977 Star Wars came out, and the phase two idea was cancelled and turned into a movie. In 1979 we got Star Trek The Motion Picture. It went massively over budget and failed to really capture the Star Trek spirit.

A much cheaper sequel was created in 1982 Star Trek II: The Wrath of Khan with Roddenberry taking more of a back seat. A film widely considered to be one of the best trek films. Four more feature films starring the original cast were made.

However in 1987 Gene Roddenberry teamed up with Rick Berman and The Next Generation was created with a brand new cast. The series ran for seven seasons set roughly a century after the adventures of Kirk and Spock.

Popularity of Star Trek soured and this led Rick Berman to help create the spin off series Deep Space 9, Voyager and lastly Enterprise along with four feature films continuing the adventures of the TNG cast. When Enterprise was cancelled in 2005, Star Trek appeared to be dead after 18 years on TV.

However in 2009 movie creator JJ Abrams recast the classic crew for the big screen. The film was big on action, but took place in an alternative timeline allowing for familiar characters to have new adventures without interfering with events depicted in the TV shows.

Two sequels to this film have been produced and are bringing Star Trek back into popularity. This has given rise to Discovery. Star Trek is returning to the TV screen and it is going to be great.

Azure Friday

funkysi1701@gmail.com (funkysi1701) — Mon, 18 Sep 2017 20:00:45 +0000

I think Azure is great, but there is loads to it so I can never know about all of its features. There is a video series hosted by Scott Hanselman called Azure Fridays which I have started to watch in an effort to keep more up to date about some of its cool features.

I watched this video recently and it is all about application insights and new ways you can debug your web applications by creating snapshots. I am a big fan of application insights so adding extra ways to debug my apps is a big win for me. Once I get this feature working in my code I will no doubt blog about it.

Azure Functions

funkysi1701@gmail.com (funkysi1701) — Tue, 12 Sep 2017 20:00:45 +0000

I recently blogged about using Azure Web Jobs , Azure Function is another way of doing the same thing, lets look at how they work.

(Sorry its been a while since I blogged but I suspect an erratic schedule will continue for the next few months.)

To create an Azure Function go to the Azure portal and click add new and search for “Function App”

Give your app a name and select the usual resource group and location settings.

Now when you open Function Apps you should see your new app.

I want my Function App to run on a schedule so I clicked the + next to functions and selected TimerTrigger. I am a c# programmer so I selected this option as well.

Give your function a name and specify using the usual cron notation how often it should run. I want mine to run at 9.30pm each night so use 0 30 21 * * *

Now comes the code bit. By default you get a window with the following code in it

using System;

public static void Run(TimerInfo myTimer, TraceWriter log)
{
 log.Info($"C# Timer trigger function executed at: {DateTime.Now}");
}

It is entirely up to you what you get your function to do. In my case I just wanted to call a URL on a schedule so I created some code that used httpclient.


using System;
using System.Net.Http;

public static async Task Run(TimerInfo myTimer, TraceWriter log)
{
 log.Info($"Buffer 0 function executed at: {DateTime.Now}");
 HttpClient client = new HttpClient();
 var result = await client.GetAsync("URL");
 string resultContent = await result.Content.ReadAsStringAsync();
 log.Info(resultContent);
}

Once you have created your app and it has run you can use the monitor section to view success and failures.

There is loads more you can do with Azure Function but this is a good place to start.

Pwned Pass available on Play Store

funkysi1701@gmail.com (funkysi1701) — Mon, 14 Aug 2017 20:00:45 +0000

Pwned Pass is now available from the Google Play Store.

Pwned Pass is a simple android app that allows you to type in a password and tells you if it has been used in a data breach.

Troy Hunt of Have I Been Pwned? recently added a new API to his website which allows you to search his extensive database of pwned passwords, 306 million of them. I have simply created a Android frontend to this API.

The API itself takes a SHA1 hash of the password and either returns a HTTP 200 if the password is found or a HTTP 404 if the password does not exist in the HIBP database. For more details of how Troy Hunt created this check out his blog post .

My app simply generates a SHA1 hash of anything that is typed in and then passes this to Troy Hunt’s API. I then get the HTTP return code so I know if the password exists or not.

It should be noted that: Do not send any password you actively use to a third-party service – even this one! I don’t log anything that you type into my app and all I am then doing is passing a SHA1 hash over SSL to HIBP. However you shouldn’t trust my word alone.

The app itself is written in Visual Studio with Xamarin Forms in a similar fashion to the app I talked about last week .

As I am using Xamarin Forms there is the potential that I may develop iPhone or UWP versions of this code in the future. With that in mind I have made use of interfaces for the android specific parts of the code.

I also make use of the modernhttpclient nuget package due to problems I encountered with httpclient and SSL. This is due to limitations of what libraries are available in mono and what has been implemented, I suspect there are better ways to solve this but that is all part of the fun.

Please do have a look at Pwned Pass and let me know what you think. Especially if it doesn’t work or throws errors. I would like to spend time making this app as good as I can make it.

Android apps with Mobile Centre

funkysi1701@gmail.com (funkysi1701) — Mon, 07 Aug 2017 20:00:45 +0000

For the past week or so I have been playing around with Xamarin and creating an android app.

Well I now have an app in the Google Play Store.

Before you rush and download the app I must warn you that it doesn’t do much yet. It displays some content that is on my website and there are a few links to allow sharing of content. I have some ideas to display content from my blog and allow sharing. I also have some other ideas for apps that might actually be useful to people that are not me. If you have ideas or feature requests do let me know.

Ok how did I go about creating this app and getting it in the app store?

Xamarin is now part of Visual Studio so step one is install all the Xamarin features to Visual Studio and build an app.

Next I wanted to monitor my app. Now I know Application Insights doesn’t support apps so what tools are out there? HockeyApp is something I had heard of but they are in the process of being replaced with https://appcenter.ms/apps. (Edit: Visual Studio App Center was retired on March 31, 2025)

It was relatively easy to hook up my app to Visual Studio Mobile Centre. First install the required nuget packages. Then add using statements and the following line to your MainActivity.cs file (these instructions are available on the Mobile Centre)

using Microsoft.Azure.Mobile;
using Microsoft.Azure.Mobile.Analytics;
using Microsoft.Azure.Mobile.Crashes;
using Microsoft.Azure.Mobile.Distribute;
using Microsoft.Azure.Mobile.Push;

MobileCenter.Start("[Unique ID]",typeof(Analytics), typeof(Crashes), typeof(Distribute), typeof(Push));

Now you can connect the Mobile Centre to your source code (VSTS in my case) and get it to run a build for every commit.

One complexity of the build is that you need to supply a keystore file (basically a certificate to digitally sign your app). I found the best way to do this was to use Visual Studio to create the file.

In VS2017 there is a option called Archive Manager under the tools menu. In here click the distribute button and select Ad-hoc. In the signing identity section you can create a keystore file. Enter a few details and a keystore file will be created in AppData\Local\Xamarin\Mono for Android\Keystore[keystore name][keystore name.keystore]

Once you have added the keystore file to your build you can enable the distribute option. Now you will get an email after every build with a link to install your app.

Every time your app crashes the details will be logged in the crashes section for you to explore and fix the issues.

The Analytics section allows you to explore how your app is being used. You can also add Analytics.TrackEvent(“Feature X”) to measure the usage of different features.

There are more things you can do which I will explore more at another time along with how to get your app into the Google Play Store.

Creating NuGet packages with VSTS

funkysi1701@gmail.com (funkysi1701) — Mon, 31 Jul 2017 20:00:45 +0000

For a while I have found myself writing the same bits of code for different web projects. This annoys me as it goes against the DRY principle (don’t repeat yourself).

One possible solution is to write your own nuget packages. You can then add this piece of code to any project you work on.

nuget.org is the public nuget feed where any developer can download nuget packages. You could publish your nuget package here, but your might want to restrict access so better to create a private nuget feed.

Lets look at how we create a nuget feed in Visual Studio. First thing you need to do is install the Package Management extension to Visual Studio Team Services (its free for less than 5 users), this will add a packages section under the build menu.

Before you can start using this new feature you need to add a Package Management License in the users hub.

Once that is done you can create a feed. You need to give your feed a name, decide if only members of the current project or everyone in your account should have access to read and contribute to.

Now you have a feed you could use the nuget package command to create a nupkg file and then nuget push command to add it to your feed. A better way is to get Visual Studio Team Services to do all the hard work.

Create a new project in Visual Studio Team Service to house your nuget package. In the build section add an empty build definition. Choose a build agent, I am using the Hosted VS2017. Then add the following steps nuget restore, Visual Studio Build, nuget pack and nuget push.

nuget restore – this step is only needed if your code depends on other packages. If it depends on other packages that are only in your feed you must specify your feed in the feeds and authentication section.
Visual Studio Build – this builds your code like you would in Visual Studio. The only config I made to this step was to specify Release in configuration.
nuget pack – this creates the nupkg file from your built project. In configuration to package specify the same as you specified in the previous step (in my case Release)
nuget push – this publishes to your feed, so of course you need to specify your feed.

One last thing to configure is to enable the continuous integration option in triggers. This means whenever you push code all these steps will run and you have a new version of your nuget package.

In Visual Studio you need to create a *.nuspec file, this contains all the meta data about your nuget package. Let look at an example.

<?xml version="1.0"?>
<package >
 <metadata>
 <id>$id$</id>
 <version>1.0.2</version>
 <title>Nuget</title>
 <authors>Simon Foster</authors>
 <owners>Simon Foster</owners>
 <requireLicenseAcceptance>false</requireLicenseAcceptance>
 <description>An example of a nuget package.</description>
 <releaseNotes>Release Notes</releaseNotes>
 <copyright>Copyright 2017</copyright>
 <projectUrl>https://[yourVSaccount].visualstudio.com/nuget/</projectUrl>
 </metadata>
</package>

One last thing to mention is version numbers. You can either change the version number in your *.nuspec file everytime you push changes. This will create stable packages like 1.0, 1.1, 1.2 etc

However you can use the automatic version number setting in the nuget pack build step. However I have found this only ever creates pre-release packages and I haven’t found a way to upgrade a package from pre-release to stable.

This is a really neat way to reuse your code in multiple projects. I have only been looking at this for a few days and I have already extracted code to do with emails, creating excel downloads and database related methods. I suspect that doing this will also have a side benefit of forcing me to create code with fewer dependencies so more code can be turned into a nuget package.

Async and Await

funkysi1701@gmail.com (funkysi1701) — Mon, 24 Jul 2017 20:00:45 +0000

For a while the Async and Await commands in c# have confused me.

Like most things the best way to learn about something is to use it in a real world example. I am currently adding an email alert feature to a website. This is an ideal example of something that would benefit from Asynchronous programming. There is no need for the webpage to wait to send 1000s of emails, lets just send a call to get started and allow the browser to carry on as normal.

This is my first try at using async and await so feel free to suggest best practises in the comments.

Lets start with a Send method in my EmailController.

public ActionResult Send(int id, int pageId, int userID)
{
 if (!Authorize.checkPageIsAuthorised(userID, (Authorize.PageIds)pageId))
 {
 return Redirect("/login");
 }
 else
 {
 Task<string> t = SendNotifications(id,userID);
 return Redirect(Request.UrlReferrer.ToString());
 }
}

This simply checks to see if you have permission to the page. If not redirects to the login page otherwise it makes a method call and redirects back to the page it came from.

Lets have a look at that method call in more detail.

Task t = SendNotification(id, userid);

SendNotification doesn’t return a normal string it returns a Task, so lets look at how we are creating this.

public async Task<string> SendNotifications(int id,string type,int userid)
{
 //logic ommitted
 await ef.SendEmail(model, emailHtmlBody);
 return "OK";
}

The return type is set to Task but it has the aysnc keyword appended to it. It also makes a call with the await keyword.

public async Task SendEmail(EmailModel model,string emailHtmlBody)
{
 //logic removed
 await smtp.SendMailAsync(message);
}

So that is it. My first bit of code that uses Async and Await. My controller calls a method asynchronously which then calls another method asynchronously which sends emails asynchronously.

Async – This enables the Await keyword to be used in the method

Await – This is where things get asynchronous. The await keyword allows the code to wait asynchronously for the long running code to complete.

How good is your SSL?

funkysi1701@gmail.com (funkysi1701) — Mon, 17 Jul 2017 20:00:45 +0000

I recently watched Troy Hunt’s What Every Developer Must Know about HTTPS course on Pluralsight. Its very good and really make you think about SSL certificates and how to correctly implement them.

One thing in particular Troy mentions is the website SSL Labs . This website allows you to test a websites implementation of SSL. A grade of A to F is assigned with A being the best and F being the worst.

Troy Hunt has a blog post where he discusses how Australian Banks score. Lets look at a few UK banks.

Bank	SSL Certificate	Grade	Home Page Under SSL
HSBC	www.hsbc.co.uk	B	Y
Nationwide	onlinebanking.nationwide.co.uk	C	N
NatWest	www.nwolb.com	C	N
Barclaycard	www.barclaycard.co.uk	A-	Y
Barclays	bank.barclays.co.uk	A-	N
Lloyds Bank	www.lloydsbank.com	A	N
Royal Bank of Scotland	www.rbsdigital.com	C	N
Standard Chartered	www.sc.com	C	Y
Virgin Money	uk.virginmoney.com	A+	Y
Santander	retail.santander.co.uk	A-	N

On the whole the ratings are all quite good with all being in the range A-C. However I have also indicated if they have SSL on the home page. Only 4 out 10 website listed above have the home page load under SSL.

Why does this matter as long as the login is under SSL? Any page that loads over http is potentially at risk from a man in the middle attack. A fake malicious home page could contain links to any page and trick users into entering personal information.

If you want to test a bank or other website not listed here. Go to https://www.ssllabs.com/ssltest/index.html and type the address that is on the SSL certificate in to the search. The good news is that this site scores a A.

Troy mentions that there is rapid growth in the adoption of SSL, there is also rapid growth in improving ratings. One of these banks went from a C to an A during the course of writing this blog.

Writing your first test

funkysi1701@gmail.com (funkysi1701) — Mon, 10 Jul 2017 20:00:45 +0000

Whenever I write a new test I have to think how best to do it. Hopefully I can summarise a few tips here to help get started.

Arrange Act Assert

The first thing I think about when writing a test is Arrange, Act, Assert. Arrange, Act, Assert is a pattern for writing the tests.

Arrange – This gets things in order ready to execute the test.

Act – This executes the method you want to test.

Assert – This compares the value produced in the Act step with a known value typically with a method similar to the following

Assert.AreEqual(expected value, actual value)

Say for example you wanted to test a method called ReturnsTrue() which does nothing but returns a value of true. This method is in a class called ReturnsTrueClass

The Arrange step in this example would be.

ReturnsTrueClass t = new ReturnsTrueClass();

The Act step in this example would be.

var result = t.ReturnTrue();

The Assert step in this example would be.

Assert.AreEqual(true, result);

This is a stupidly simple example but hopefully you get the idea of how you can build all your tests with these three steps.

Recently I saw a tweet complaining that someone has mixed up expected and actual in the Assert statement.

There is a minor but special hell reserved for those who mix up the expected and actual parameters in Assert.Equals
— Keith Williams (@zogface) July 5, 2017

At first glance this probably isn’t the worst mistake to make as if your tests are all passing actual and expected are the same.

However tests will fail, that is the whole point of them, you can then fix bits of code. If you have mixed up actual and expected it adds extra time to debugging and figuring out what values are produced from your code and what you are expecting it to produce. It may be your test uses a mocking framework and somewhere in there, there is an issue, with mixed up expected/actual you may assume a problem in your code rather than the test.

Also, how do you make such an error? When I type Assert.AreEquals() in Visual Studio, Visual Studio tells me what each parameter does, it takes a matter of seconds to do this, just by hovering over the code.

One last tip to say about tests. Write your tests to test the behaviour of your application.

Uploading Files to AWS

funkysi1701@gmail.com (funkysi1701) — Mon, 03 Jul 2017 20:00:45 +0000

I am a fan of Azure but today I have been looking at AWS. Specifically how to upload and download files.

AWS S3 stores files in Buckets. I already had an AWS S3 account setup with a Bucket. I am going to assume you have got a bucket setup and concentrate on the code to get files in and out.

First step is to use nuget to install the AWS packages. In nuget the packages you want are called AWSSDK.Core and AWSSDK.S3.

The using statements you want to use are called Amazon.S3 and Amazon.S3.Transfer, not sure why this doesn’t match nuget, this difference caught me out a couple of times.

Now to the code that uploads files

AmazonS3Client AWSclient = new AmazonS3Client(accessKeyID, secretAccessKeyID, Amazon.RegionEndpoint.EUWest1);
TransferUtility fileTransferUtility = new TransferUtility(AWSclient);
using (FileStream streamWriter = new FileStream(path, FileMode.Open))
{
 TransferUtilityUploadRequest fileTransferUtilityRequest = new TransferUtilityUploadRequest
 {
 BucketName = "flawlessimages",
 InputStream = streamWriter,
 Key = fileName
 };
 fileTransferUtility.Upload(fileTransferUtilityRequest);
}

Lets break it down and look at what it does.

AmazonS3Client AWSclient = new AmazonS3Client(accessKeyID, secretAccessKeyID, Amazon.RegionEndpoint.EUWest1);

This creates an instance of AmazonS3Client, we are passing the Access Key and Secret Access Key both of which can be found from your Amazon S3 account My Security Credentials section. Amazon.RegionEndpoint.EUWest1 specifies the amazon data centres that your bucket is located in.

TransferUtility fileTransferUtility = new TransferUtility(AWSclient);

This creates an instance of TransfterUtility using the AmazonS3Client instance we created in the previous step.

using (FileStream streamWriter = new FileStream(path, FileMode.Open))
{

This opens up a filestream from a files path and specifies that the file should be opened.

TransferUtilityUploadRequest fileTransferUtilityRequest = new TransferUtilityUploadRequest
{
 BucketName = "flawlessimages",
 InputStream = streamWriter,
 Key = fileName
};
fileTransferUtility.Upload(fileTransferUtilityRequest);

This last step specifies which bucket to upload to, what input stream to upload and the Key to use. Key is just AWS way of referring to files, more commonly referred to as the filename.

This is all you need to do to upload a file to your Bucket. The file will be located at https://s3-eu-west-1.amazonaws.com/[bucketname]/[filename] , however by default it will not be downloadable until you set Read permission to everyone, once you do that anyone who has the link will be able to download your file.

This is the same permission level as any file you have on your webserver, however AWS has a better way.

using (s3Client = new AmazonS3Client(accessKeyID, secretAccessKeyID, Amazon.RegionEndpoint.USEast1))
{
 GetPreSignedUrlRequest request1 = new GetPreSignedUrlRequest
 {
 BucketName = bucketName,
 Key = filename,
 Expires = DateTime.Now.AddMinutes(5)
 };
 urlString = s3Client.GetPreSignedURL(request1);
}

Here we are generating a url to download the file, but we are specifying that it is only valid for 5 minutes. This means that if you share the url it will only work for 5 minutes, after that AWS will give an access denied message.

This is much better security than you have on a typical web server, and easy to implement, every time a user clicks on a download link you generate a new presigned url and send the download to the browser, as long as this process doesn’t take longer than 5 minutes the user will never know.

Using Azure WebJobs to automate

funkysi1701@gmail.com (funkysi1701) — Mon, 26 Jun 2017 20:00:45 +0000

I keep hearing about Azure WebJobs but I have never used them. Time to change this.

WebJobs are a feature of Azure App Service that can run a script at a specific time. In my case I would like to hit a specific url of my website at the same time every day.

To the right you can see an example of the WebJobs form on the Azure portal that you need to fill in.

You need to supply a name for your webjob.

You need to upload the script that will run in my case I used a powershell script. My script consisted of which basically just loads the url specified.

$progressPreference = "silentlyContinue";
$result = Invoke-WebRequest -Uri ("https://www.google.com") -Method Get -UseBasicParsing;

Type refers to if your job will be triggered or run continuously, I want it to be triggered.

Triggers refers to if you want it to be scheduled or manual, something that you can run on an ad hoc basis. I of course want scheduled.

If you are familiar with the linux CRON then the next box will make sense to you for everyone else I will try and make sense of it. The box consists of 6 numbers which can either have a value or a *. The numbers correspond to the following {second} {minute} {hour} {day} {month} {day of the week}.

A hourly job would be expressed as 0 0 * * * *, ie every day of week, every month, every day, every hour and only when minute and second equals zero. For more help with this check out the MSDN docs about it. I want to use 0 30 21 * * * to run daily at 9.30pm.

That’s it everything setup, now time to wait and see if it works.

Oh no!

It failed to run at the specified time.

The reason for this is the scheduler requires the feature Always On to be turned on which is not available in the free App Service. Before you reach for your wallets, I found a solution on this blog post that allows them to run on the free tier.

The thinking behind this solution is you need to keep the website alive throughout the day so Tom has created a script that does this. His script can be found on his blog or on his github page .

Set this script up to run every 5 minutes (0 */5 * * * *) like the example above.

The nextthing you need to do is create a Custom connection string in the Application Settings blade called SecretThing. Tom’s script references this to access the website and keep it alive. The password you need to put in SecretThing can be found in you publish profile (downloaded from the Overview blade in the Azure portal). For more details and a better explanation check out Tom’s blog .

One last thing to mention about WebJobs is that you can see details about when they have run at https://[YourWebAppName].scm.azurewebsites.net/azurejobs/#/jobs and this can be a great place to help debug your scripts.

Fiddler and APIs

funkysi1701@gmail.com (funkysi1701) — Mon, 19 Jun 2017 20:00:45 +0000

A while ago I blogged about promoting my blog with Buffer. At the time I made use of the nuget package BufferAPI but lets look at some improvements I can make.

The BufferAPI package worked great from my console app, but when I tried to use it from a Controller in an MVC app I never got it to work. Lets look at the API docs and see if I can rewrite it.

There are two main types of API calls GET which gets data from the server and POST which posts data to the server. These come from the types of HTTP requests.

I quickly figured out how to use the GET API call to authenticate using https://api.bufferapp.com/1/profiles.json?access_token=XXXX

However POST was defeating me. That was until I remembered Fiddler .

I had heard Troy Hunt (and others) talk of using Fiddler to examine what data is being passed among websites. Troy uses it to do a man in the middle test to see what information can be stolen.

It is really easy to setup, install Fiddler, click yes to a few security warnings and you can see what information is being passed from your code to remote APIs.

Once I had Fiddler installed I could compare what information is being passed between a successful API call using the BufferAPI nuget package and an unsuccessful API call using my code.

Fiddler also showed that passing my authentication token in a POST request is much better. Despite both GET and POST being encrypted when using HTTPS, anything at either end that logs URLs will have a log of your username and password.

If you have not tried Fiddler, give it a try especially if you are doing things with API calls.

Add Power BI data to a webpage

funkysi1701@gmail.com (funkysi1701) — Mon, 12 Jun 2017 20:00:45 +0000

Last week I talked about Power BI , what it is and some of the different services you can use with it. This week lets add some of that data to a simple web page.

For this example I am going to add the google analytics data from this website to this page.

First login to your Power BI https://app.powerbi.com/

Click the get data link at the bottom left.

Click My Organisation to bring up the app search box.

Click the Apps tab and search for “google” in the search box, you should then see Google Analytics, click into this and then click the get it now button.

Log into your google account. If you have multiple google accounts I found it worked best to sign out of all of them or run this in an incognito window.

Once you are signed in you should see a list of the different google analytics data you have, select the one you want to use and click import.

Power BI will then go away and start loading the data.

Once loaded go to Reports and select the Google Analytics that has been loaded. If you have more than one, it is a good idea to rename each one eg Corporate Site Google Analytics, Blog Google Analytics so you won’t get mixed up.

In the file menu select Publish to web and agree that you are OK for this to be made public.

You will then be given a piece of HTML code that starts with <iframe copy this onto your web page. Reload your webpage and you should see something similar to mine below.

It should be noted that while you can but the webpage containing the iframe behind a login page, the data could still be accessed if you knew the url contained within the iframe, this is why the link can be emailed and continue to work.

What is Business Intelligence here?

funkysi1701@gmail.com (funkysi1701) — Mon, 05 Jun 2017 20:00:45 +0000

The term Business Intelligence (BI) refers to technologies, applications and practices for the collection, integration, analysis, and presentation of business information. The purpose of Business Intelligence is to support better business decision making. Essentially, Business Intelligence systems are data-driven Decision Support Systems (DSS). Business Intelligence is sometimes used interchangeably with briefing books, report and query tools and executive information systems.

What is Power BI?

Power BI is a business analytics service provided by Microsoft. It provides interactive visualisations with self-service business intelligence capabilities, where end users can create reports and dashboards by themselves, without having to depend on any information technology staff or database administrator.

Microsoft describe it as follows: Power BI is a suite of business analytics tools that deliver insights throughout your organisation. Connect to hundreds of data sources, simplify data prep, and drive ad hoc analysis. Produce beautiful reports, then publish them for your organisation to consume on the web and across mobile devices. Everyone can create personalised dashboards with a unique, 360-degree view of their business. And scale across the enterprise, with governance and security built-in.

What can you do with it?

One of the first thing you can do with Power Bi is connect some of the apps that you use.

To sign up to use Power BI go to the Power BI website .

Once you are signed in click the get data link in the bottom left corner and browse through the apps you can use.

If your website uses google analytics you can connect it up to Power BI and see some cool analytics. The below map was generated from some google analytics data, you can see that this website has had users across the globe, with UK and USA getting a lot of hits.

Another thing you can do is connect Azure Application Insights. This provides some similar data to google analytics about who is visiting your site.

If your code is hosted in Visual Studio Team Services you can get some cool analytics from your commit history.

There is a huge long list of apps you can connect to power BI so you can start getting all sorts of analysis very easily. I am hopefully going to talk another time about getting custom data into Power BI.

Have I Been Pwned?

funkysi1701@gmail.com (funkysi1701) — Mon, 29 May 2017 20:00:45 +0000

Have you heard of the website https://haveibeenpwned.com ?

Well you should have. Have I Been Pwned is a website created by security expert Troy Hunt that keeps track of data breaches and allows you to search and find ones that affect you. As I write this Troy Hunt has tracked 3,752,984,562 pwned accounts from 216 pwned websites.

What does “pwned” mean?

pwn means to compromise or control, specifically another computer (server or PC), website, gateway device, or application. (as defined on wikipedia)

Originally, pwn and its variants were pronounced /ˈoʊn/ in the same way as the verb own, the tail of the p being “silent”.

In terms of this site Troy Hunt defines it as:

A “breach” is an incident where a hacker illegally obtains data from a vulnerable system, usually by exploiting weaknesses in the software. All the data in the site comes from website breaches which have been made publicly available.

What can I do?

As an individual you can search for your email address in Have I Been Pwned, I am in 7 data breaches.

As a person responsible for an email domain you can search and find which of your users are in a data breach.

You can also sign up to get notified of any future data breaches you might get caught up in.

If you are in a data breach change your password. If you use the same password across multiple sites or services change them as well. Consider using a password manager so you can have unique complex passwords for every services you use and not have to worry about forgetting them.

If you look after a website or service then follow Troy Hunt. Think about security, is your site vulnerable to SQL injection, do you store passwords with reversible encryption?

How worried should I be?

Data breaches are happening more and more often. Its not showing any sign of slowing down, in fact I expect there to be lots more in the years to come. However there are things you can do to mitigate the damage of being in a breach.

Use a password manager
Don’t share passwords between sites
Regularly change your passwords
Think of passwords as pass phrases and include spaces between the different words

Consider what information a company has about you. How worried would you be if this became public knowledge? Consider if you want this online, weigh up the benefits etc

Star Trek Discovery trailer thoughts

funkysi1701@gmail.com (funkysi1701) — Mon, 22 May 2017 20:00:45 +0000

I am really excited a new Star Trek Discovery trailer has just been released, if you haven’t seen it yet, watch it now.

The rumour mill has been going crazy about Star Trek Discovery but now we have a trailer with actual footage to analyse. In my opinion all the concerns about changes in the production team have gone out the window, this looks to be a polished Star Trek show that can sit alongside our favourites.

The Ship

When images first surfaced of the ship I wasn’t keen, however all the shots of the ship look great. I assume this must be the USS Discovery, although it is hard to tell anything from a two minute advert.

The Bridge

The Bridge looks recognisable as a Star Trek bridge, but also looks fresh and new. I think it feels very JJ Verse. I like it.

The Uniforms

The uniforms look like a mid step between what we saw on Enterprise and what we know as the Classic Trek look. I can’t see any Blue, Red, Gold to signify departments, but it looked like Gold and Silver could well signify something. Really hard to tell at this stage, but I like what I can see.

Characters

The show feels like it is going to be gritty and real. With characters you can really get behind. I liked what I saw of the captain and lead character. The alien character intrigued me, hopefully his line about death will be explored. It feels like there is a strong link between Vulcans and Sarek with the shows lead (played by Sonequa Martin-Green) Not sure how I feel about this yet, depends on what they do with it. I don’t want them to just redo what they have done with Spock, but again lets wait and see.

Klingons

OK I am not sure I like the strong emphasis on the Klingons. I think we have spent a lot of time learning about Klingons, so when I heard the casting news about Klingons I was not excited. The Klingons also have a new look, which doesn’t bother me that much as the Klingons have changed their look before. But lets wait and see what happens.

Star Trek is back

After over a decade with no Star Trek on TV, it is finally coming back. I am beyond excited about this. There are lots of things I want from this new show but this advert has definitely rekindled my excitement and hope that we are going to get something good.

Windows Update – don’t turn it off!

funkysi1701@gmail.com (funkysi1701) — Mon, 15 May 2017 20:00:45 +0000

No blog post this week. However why not check out Troy Hunts blog article about Windows Update .

I am probably fairly unique in that I like installing updates whether it be on my phone, my PC or for one of my favourite programs. One of my tasks when I used to work in a sysadmin role was ensuring all windows updates got installed on all servers and client machines. It was a never ending task, as soon as you got almost all updates done, it would be update Tuesday and a new set of updates would be released.

I can think of a couple of times an update got installed that caused a problem and needed to be rolled back. However I have no way of measuring how often it fixed a problem or improved the security of the OS or other software. If you compare the annoyance of waiting for updates to install against having a computer (or worse a server) be hacked or otherwise compromised I think it is fairly clear what you need to do.

As someone who works in IT I feel it is my duty to share the importance of keeping your devices updated. Don’t turn automatic updates off or disable it. It is there to help you and keep your computer safe and secure.

HighCharts

funkysi1701@gmail.com (funkysi1701) — Mon, 08 May 2017 20:00:45 +0000

High Charts is a javascript library that allows graphs and charts to easily be added to web pages.

A chart like this can easily be added with a few lines of html and javascript.

<script src="https://code.highcharts.com/highcharts.js">
<script src="https://code.highcharts.com/modules/exporting.js">
<div id="container"></div>

This adds the highcharts library and a container for the chart to be displayed in.

Highcharts.chart('container', {

 title: {
 text: 'Solar Employment Growth by Sector, 2010-2016'
 },
 subtitle: {
 text: 'Source: thesolarfoundation.com'
 },
 yAxis: {
 title: {
 text: 'Number of Employees'
 }
 },
 legend: {
 layout: 'vertical',
 align: 'right',
 verticalAlign: 'middle'
 },
 plotOptions: {
 series: {
 pointStart: 2010
 }
 },
 series: [{
 name: 'Installation',
 data: [43934, 52503, 57177, 69658, 97031, 119931, 137133, 154175]
 }, {
 name: 'Manufacturing',
 data: [24916, 24064, 29742, 29851, 32490, 30282, 38121, 40434]
 }, {
 name: 'Sales & Distribution',
 data: [11744, 17722, 16005, 19771, 20185, 24377, 32147, 39387]
 }, {
 name: 'Project Development',
 data: [null, null, 7988, 12169, 15112, 22452, 34400, 34227]
 }, {
 name: 'Other',
 data: [12908, 5948, 8105, 11248, 8989, 11816, 18274, 18111]
 }]

});

This adds the data and sets up various options for the charts. More details about the different charts and options you can set can be found at https://www.highcharts.com

I am quite a fan of the different gauges that you can put on your site, as you can add a bit of animation and make the needles bounce around. https://www.highcharts.com/demo/gauge-solid

As most of the data I deal with lives in a SQL Server database, I have been spending quite a bit of time writing Stored Procedures and functions to mould the data so I can easily pass data from the database to javascript.

Highcharts are very flexible with lots of different options that allow you to display charts of almost any data you have, these charts are also exportable as PNG, JPEG, CSVs etc

Office Move

funkysi1701@gmail.com (funkysi1701) — Mon, 01 May 2017 20:00:45 +0000

For the past few weeks my software developing has been taking a back seat as I planned and coordinated the IT requirements of an office move.

The company I work for has been working out of a converted barn, but as the company has grown we have outgrown the building and for quite a while it has been a real squeeze to get everyone in. We now have a shiny new offices with plenty of room for growth.

It was at the start of the year when I first started getting involved with the new office. At that time building work was about to begin. The office had concrete floors so in order to get network and power points in the floor, channels would have to be cut into the concrete for the floor boxes and cable runs.

At the old office our servers were in a cabinet in the corner of the room which meant they could be quite loud especially when the fans were going full pelt. Our American owners were supplying brand new IT equipment and we would have a dedicated server room.

Soon a weekly conference call was setup so we could coordinate with the IT people in America and the various different contractors that would help deliver our new office.

One thing I was particularly proud of was YDS. York Data Services (YDS) is an ISP I have worked with in the past at a previous job and I was able to continue my relationship with them and they were contacted and became our primary internet supplier.

Unfortunately most ISPs have to deal with BTOpenreach who have a bit of a monopoly on getting phone lines or leased lines installed. The initial estimate we were given was two weeks after we were due to move in. It was looking like we would move in and two weeks later we would get an internet connection. We were in the process of getting quotes for a temporary internet connection when BT contacted us and could get us connected up.

Soon afterwards a huge cabinet and two pallets full of equipment was delivered. Two days later a team was dispatched to rack everything up. On the rack we had the following equipment: 4 x UPS, 4 x Network Switches, 2 x Routers (for 2 x Leased Lines from different providers, the secondary connection has not been installed yet), 2 x Palo Alto Firewall devices, 2 x New Servers.

Just prior to our move we had a rack full of equipment but no patch cables, nothing connected up and I was beginning to get concerned that we would not be ready in time.

Thankfully a team was dispatched to help out over the weekend of the move, with the patch cables being delivered only hours beforehand. I worked closely with them and we worked late into the night. By the end of the Friday night our domain controller had been moved and its IP address updated. DHCP scopes had been setup for the new network. Once that was done we could get the WiFi points connected up (These had already been fitted to the ceiling).

The following day we moved our existing servers and updated their IP addresses and got everything patched up. Furniture started arriving so desks could start being setup and phones connected up. Everything was slotting into place, and for the first time our new office was starting to look like an office.

By the time it got to Monday the only problem was the phone number had not transferred as requested (something else to blame on BT) All staff could either connect via an Ethernet cable or connect to our brand new WiFi network and had access to all the IT services they had at the old office. Another minor issue was the NAS we used for backups had to be reset as it couldn’t communicate with the domain so no one could login, luckily this didn’t affect the data on it.

From my perspective the move was a massive success considering how complex and how many different people had been involved.

GitHub history

funkysi1701@gmail.com (funkysi1701) — Mon, 24 Apr 2017 20:00:45 +0000

I created my GitHub account in August 2010, lets look at what I have done with it.

In 2010 I started by committing the code for an old php website I had created. Then in December I committed some other php sites.

Between 12 December 2010 and 29 September 2011 I must have created some automated process which is why there is a commit every day. Looking at the diff it appears to be related to tracking visitors to a site. I assume back then I hadn’t heard of the .gitignore file!

Nothing for a few years until 2014, when I started compiling a code samples collection, this is bits of code that I want to show off, I have added to this since 2015 so I really should go back to this as my skills have developed a bit since then.

In 2015 I really started to think of myself as a developer, I added repositories for Raspberry Pi, and a few C# ideas I had. I ended the year taking part in Advent of Code

What did I do last year? Well I started going to York Code Dojo and this meant lots of forks from their code examples and also my first pull request. I also did a fair bit of looking at other repositories trying to find a good open source project to contribute to, still not found one for me yet. Lastly I ended 2016 with a bit of Advent of Code.

Not sure how enthralling a blog post this is, but fascinating to see the different commit histories.

Automating promotion of my blog

funkysi1701@gmail.com (funkysi1701) — Mon, 17 Apr 2017 20:00:45 +0000

I would like to automate the promotion of this blog. Currently to promote this blog on social media I use a few different services.

Buffer (buffer.com ) is a service that allows you to schedule updates to the main social media channels (Facebook, Twitter, LinkedIn and Google+)

IFTTT or If This Then That (ifttt.com ) is a service that allows you to connect different online services. You can send an email when a specific event occurs in your calendar for example.

I have been using a combination of these services to share to social media some of my past blog posts. I then add to my calendar details of my blog posts. Then I use IFTTT to add the event to Buffer, and then buffer tweets on a schedule.

This works great however it is a manual process to add my posts to my calendar. I have been using a spreadsheet to help me generate an ics calendar file which I import into my google calendar. There must be a better way of doing this.

I have written some code that reads the RSS feed of my blog and then shares that to Buffer using the Buffer API. The code I am creating is far from finished however I am trying to use the concepts of clean code to make it as flexible as possible.

I have an interface called ISocial which my buffer code implements, but it would be easy to add a class that implements the same interface but uses the twitter or facebook APIs. My code reads from a specific WordPress RSS feed, but it should be easy enough to adapt to read from a SQL database or any other data source.

I am currently unsure what kind of interface to use for this application. I could create a web page that controls the functionality, or maybe just a windows application, or maybe both of these are complicating things and all I need is just a console application that could be added as a scheduled task.

Blame

funkysi1701@gmail.com (funkysi1701) — Mon, 10 Apr 2017 20:00:45 +0000

I recently came home from a busy day of work to my wife blaming me for allowing our 18 month old son to reach some paint and get it all over the carpet. She was concentrating on the fact that he could reach the paint not on the fact that she had brilliantly got the paint out of the carpet.

My instinct on being blamed is to deny it, or start throwing blame back at her. It is possible I put the paint in his reach, it is also possible it was someone else.

It is very easy to blame someone else. For example, looking through the git history to find out who changed a specific file is only a few clicks.

But is it ever productive to blame someone? Is it not better to focus our energy on fixing the issue at hand?

In the world of business it is easy to go from blaming other people until you have a blame culture. When you have a blame culture everyone starts looking out for themselves so it’s not them that gets the blame and productivity will suffer.

I think it is more important to put in place processes to minimise issues happening again. If a deployment causes downtime, don’t ask who’s fault is it and sack them. Instead what can we do to reduce the chance of it happening again?

As someone who has an interest in DevOps, I often break things and don’t want to get blamed for that, I do want to improve my processes so I am not always breaking the same things.

What do you think about Blame? Is what can we do differently always better than who did this?

Interfaces

funkysi1701@gmail.com (funkysi1701) — Mon, 27 Mar 2017 20:00:45 +0000

I am trying to understand interfaces and when to use them in my code.

An interface defines a contract and any class that implements that interface agrees to fulfil that contract.

Lets look at an example as this tends to be how I learn best.

Most applications require some sort of data to work from so lets start by defining IData which can load data.

public interface IData
{
 Blog LoadData(string Connectionstring);
}

My interface defines one method LoadData and outputs an object called Blog (I will explain why in a minute)

A common data source is a SQL database so we could define a SQL class that implements IData.

public class SQL : IData
{
 public Blog LoadData(string Connectionstring)
 {
 Blog blog = new Blog();
 using (SqlConnection con = new SqlConnection(Connectionstring))
 {
 con.Open();
 //etc

We could also get data from an RSS feed of a blog (hence why I called the object Blog earlier)

public class XML : IData
{
 public Blog LoadData(string Connectionstring)
 {
 XmlDocument myXmlDocument = new XmlDocument();
 myXmlDocument.Load(Connectionstring);
 Blog blog = new Blog();
 foreach (XmlNode RootNode in myXmlDocument.ChildNodes)
 {
 //etc

Both classes implement IData and have a method called LoadData which has a string parameter and outputs a blog object. The string parameter is either a connection string to a SQL database or the URL of the rss feed. Not sure if there is a better way of doing this bit, maybe the name of the string needs making more generic.

Now we have some classes that implement an interface what can we do with them. Lets write a class called GetData which gets data but doesn’t care if it comes form the rss feed or a SQL database.

public class GetData
{
 private IData _repo;
 public GetData(IData repo)
 {
 _repo = repo;
 }

 public Blog LoadData(string Connectionstring)
 {
 var original = _repo.LoadData(Connectionstring);
 return original;
 }
}

When we call GetData we can either pass in XML or SQL as the class is not tied to either implementation. We could even write other classes that implement IData for testing purposes.

My full code can be found on github .

The advantages of writing code in this way include code that is easier to extend, easier to test and easier to maintain. This is only the start of my understanding and I am sure this is going to be a topic I come back to in the next few weeks.

SOLID and other programming terms

funkysi1701@gmail.com (funkysi1701) — Mon, 20 Mar 2017 20:00:45 +0000

This week I have been looking at improving my understanding of a few programming terms, like SOLID and I will try and define them so I can look back here when I get confused.

MVC

I have previously blogged about MVC , but my understanding was not 100% correct so I will refine this here.

Model – Now this is where my understanding was not quite correct. I thought the model was the actual source data, eg an XML file, SQL database etc. The model is the business logic so this is a processed version of the source data. MVC does not care where data is stored it can be flat files, SQL, XML or anything really.

View – This displays data to the user and typically is the HTML/CSS markup. Only display related logic would get included in the view.

Controller – This is another place I had got a bit confused. I had thought all the logic lived here. This is incorrect controllers are only concerned with getting data between Model and View.

SOLID

SOLID are five principles of object oriented programming and design.

S is for Single Responsibility Principle. A class or piece of code should be responsible for doing one thing.

O is for Open/Closed Principle. Code should be open for extensions but closed for modifications. Often this refers to the way you can implement an interface and add extra functionality.

L is for Liskov Substitution Principle. Objects in a program can be replaced with subtypes of that object with out changing functionality.

I is for Interface Segregation Principle. Large interfaces should be split down into small interfaces so that clients only know about methods that are of interest.

D is for Dependency Inversion Principle. High and Low level modules should depend on abstractions.

I need to look into SOLID some more but here are some examples .

DRY

DRY is simply Don’t Repeat Yourself. It is often easy to spot when a function call will help you not have to use the same bit of code in different places. I am often spotting places where I can put this into action.

Test Driven Development or TDD

funkysi1701@gmail.com (funkysi1701) — Mon, 13 Mar 2017 20:00:45 +0000

A few weeks back I attended a talk at Agile Yorkshire about Test Driven Development or TDD by Dr Oliver Shaw. I was impressed at how easy Oliver made it look, so as I have never tried it I thought I should give it a try.

Test Driven Development or TDD is a way of development which starts with writing a Unit Test. First you write a failing test, then you write the code to make it pass, then you refactor your code. This can be remembered by thinking of Red, Green, Refactor. Red being the failing test, Green being getting the test to pass, and Refactor being the refactoring.

During the demonstration Oliver used a language called scala and a system that automatically reran all the tests after every change. I code with Visual Studio in C# is there a way I can get my tests to run automatically as well?

A bit of googling and configuring I can answer this as Yes.

The nuget package called Giles is a watcher which will rerun your tests similar to how Oliver did it with his scalar environment. Fans of Buffy the Vampire Slayer will get the joke of why a watcher is called Giles. I couldn’t get this to work with MSTest but works fine with NUnit. There is a powershell script giles.ps1 which you need to run and will update every so often with how many tests have passed or failed. However you may not see this if you are coding in Visual Studio but there is a way to get a notification.

If you install the application Growl you can get notifications from Giles which pop up and then disappear. So whatever you have on screen you can find out almost instantly if you have broken tests.

Another thing that I wanted to configure is a way of viewing code coverage and which methods are tested and which aren’t. If you are familiar with VSTS after a build it gives you a percentage score for test coverage. I don’t find this overly useful as it doesn’t tell you what is covered and what isn’t. Also what if you want to use Github, how do you calculate the code coverage then?

The nuget packages OpenCover and ReportGenerator allow a html report of code coverage to be produced. I created a batch script that can be run whenever you require this information.

[path]\OpenCover.Console.exe -target:"[path]\nunit3-console.exe" -targetargs:"[path]\Test.dll" -output:"[path]\coverage.xml" -register:user
[path]\ReportGenerator.exe "-reports:[path]\coverage.xml" "-targetdir:[path]"

The commands are fairly straightforward, the only tricky bit is sorting out all the filepaths to the different programs.

Now that I have all this plumbing setup time to give TDD a try and see what I can build.

GitHub, Bitbucket, or VSTS compared

funkysi1701@gmail.com (funkysi1701) — Mon, 06 Mar 2017 20:00:45 +0000

As a developer using source control and git is bread and butter of what we do. Github is probably the most popular and widely known hosting service for source control but I have also used Bitbucket and Visual Studio Team Services. Lets have a look at each one and what they offer. Note while I have included prices I have only tried out the free versions.

Github

URL: https://github.com/
Private Repositories: Not Available for free
Public Repositories: Unlimited
Team Size: Unlimited
Prices: $7 per month for unlimited private repositories, $25 per month for 5 users then $9 per month per user

This is probably the most widely used service for hosting code. Over 13 million repositories of code. This is an ideal solution if you want your code to be publicly viewable, but take care not to publish passwords, private keys or your companies trade secrets. Every developer should have a Github account for displaying bits of code they are proud of.

There are a number of externally built APIs that link into Github for doing extra features, like building, code coverage etc

Bitbucket

URL: https://bitbucket.org/
Private Repositories: Unlimited
Public Repositories: Unlimited
Team Size: Less than 5
Prices: $10 per month for 10 Users, $100 per month for 100 Users, $200 per month for Unlimited Users

At my last job we used Bitbucket extensively for all our projects. All the code was private so only the team could access it, however before I left we were approaching the 5 user limit (but looking at these prices cost seems very reasonable)

Visual Studio Team Services

URL: https://azure.microsoft.com/services/devops/
Private Repositories: Unlimited
Public Repositories: Not Available
Team Size: Less than 5
Prices: $30 per month for 10 users, and other features paid for via Azure Invoices

Visual Studio Team Services or VSTS is Microsoft’s version control solution and I have only just started using it, however what I have seen I like. There are lots of options for building your code so VSTS is more than just hosting your code it is verging on a Continuous Integration/Delivery solution. Being a Microsoft product there are numerous links to Azure and it is easy to deploy stuff to that platform.

All three have options for tracking issues but VSTS have options to add Stakeholder users which would allow none developers to add and keep track of issues and progress with them.

If I want to run tests, look at code coverage VSTS is probably the solution I would go for, if I want something that is public I would go for Github. What do you think which of these is your favourite?

Learning R

funkysi1701@gmail.com (funkysi1701) — Mon, 27 Feb 2017 20:00:45 +0000

Today I spent some time learning the R language.

The problem I was trying to solve was to convert local prices of some items into Euros. I had been using a fixed exchange rate for all data, but as exchange rates fluctuate so much this is incorrect.

My first though was to find a free API that I could query to get the values I wanted. The first API I found didn’t cover all the currencies, the next one I found I burnt through the free allowance in one pass.

A colleague of mine mentioned using R to solve this, he sent me some links and I set out to write my first piece of R code.

My finished code can be found on github and I will attempt to explain some of it.

R defines functions fairly simply

nameoffunction <- function(arg1, arg2)
{
 arg1 * arg2
}

I have created a function that takes 2 parameters date and currency. I know I have about 10 different currencies that I want to get currencies for and I want to loop through each day so I will need to pass in a date.

The source of my exchange rate information is the www.xe.com website, its historical exchange rate page passes currency and date into the query string so I should be able to build up a string containing all the different elements.

All programming language can concatenate strings and R is no different. R uses paste()

var <- paste("Hello", "World")

However R has an annoying feature in this function. I would expect that var in the above example would contain “HelloWorld”, it doesn’t it contains “Hello World”. Why it automatically adds a space I don’t know?

var <- paste("Hello", "World", sep="")

I am not entirely sure what all of the code does but I can take a good guess.

read_html() I would guess loads a html page, html_nodes() finds all the html tags of a certain type on the page, in my case <table>, html_table() reads the first table it finds.

table1[2] selects the second column, and head() selects a specific number of rows. I want the first row and second column so I combine these two as head(table1[2],1)

Now that I have found my exchange rate what do I do with it? R can read and write to SQL Server so why not store this info in a SQL lookup table. I can then use this data in a stored procedure when I process my data.

To query sql you can use sqlQuery(), it has two parameters, a sql connection and a TSQL command (eg a SELECT, INSERT, UPDATE statement)

I use a while loop to loop through every day between 1st October 2016 and today and look up the exchange rate for each currency.

For now I am manually running this R script, however there are ways to run R directly from SQL Server which I may well investigate. I could then have a SQL job to run this on a schedule, maybe once a day to get the latest exchange rates. I also would like to do something a bit cleverer like only getting exchange rates for the days that I need them by querying existing database tables.

Side Project – Connect 4

funkysi1701@gmail.com (funkysi1701) — Mon, 06 Feb 2017 20:00:45 +0000

Hello?

You have probably like me almost forgotten about this blog. Life has got in the way over the last few months, but lets see if I can restart my blogging habit.

In 2017 I want to start a side project for a few reasons. I want to improve my coding skills and look at things I wouldn’t normally during my day job.

I have been trying to decide what to build. The actual project doesn’t matter too much, it is the techniques I use during build that matters the most. I would like something that I can start off as a windows app and then extend onto the web or mobile apps.

I started off thinking about a Rubik’s cube app, however my initial coding has lead me to believe it is probably too complex for my first app.

Last year I spent an evening looking at Connect 4 during a York Code Dojo session. I think this should be complex enough that I can code some logic to efficiently solve the game, however not too complex I get defeated before building too much.

So far I have built a console app that displays via Console.Write() commands, a grid to represent the game, with 0 meaning empty, 1 red and 2 yellow etc. I am currently working on the logic to solve the game, once I have that in a good state I can extend into a winforms app, into a web (MVC most likely) app and finally a mobile app (I have no knowledge about how to do this yet, but something I would like to try one day).

Over the last few days I listened to .NetRocks where they discussed Progressive Web Apps . This sounds like a great challenge for me to aim for. A progressive Web App is a website that automatically detects various properties to give you an app that runs well whatever the state of your internet connection and whatever browser you are in.

I have lots to learn but I can split it down into small sections and I can build something that demonstrates some of my skills.