2020 on Funky Si's Blog (Dev)

Set and achieve 2021 developer goals

funkysi1701@gmail.com (funkysi1701) — Mon, 28 Dec 2020 00:00:00 +0000

I didn’t make any goals for 2020, or if I did, I didn’t officially announce them. 2020 has been a hard year for all of us, 2021 is going to be better.

Here are a few ideas for my goals for the year ahead.

Azure certification
Mentoring
F#
Cosmos db/Mongo db
Give a talk
Mandlebrot Generator
Pwned Pass Mobile App
More time for me

I want to get a certificate to show how much I know. The obvious area for this is Azure. I spend a lot of time playing with Azure, building and deploying to the platform I should be able to get certified in this area. I actually had an exam booked in 2020 but it was an in-person exam so was cancelled when Covid hit. I plan to sit the foundation exam in the first quarter of 2021 and I will take it from there.
I have had a few times recently where I have been reminded that I don’t have much experience working with others. Mostly because I work in a one-man development team so there are limited opportunities in the workplace. I have a few ideas to change this, but this is one of my top priorities for 2021.
& 4) My experience is very Microsoft and C# so I want to spend some time exploring and learning tech that is adjacent to this. A functional language like F# sounds like a good compliment to my existing skills and my data skills are also very SQL server so some document database skills would be a good place to spend some time.
Doing a talk has been on my list for years. I am terrified of doing one and with everything online now I don’t know if that makes things easier or harder. I don’t think 2021 is going to be the year for this but maybe I will take baby steps towards this goal.
Twitter recently reminded me about watching a Mandlebrot generate one pixel at a time in the 1980s. I would love to explore the code used to generate them and how fast they currently are to produce.
My Google play store Xamarin forms app that uses the HIBP API has less than 100 users and I am considering closing it down, especially as I pay a monthly fee for API keys. I haven’t really decided what to do, maybe I will spend some time improving it, maybe I will close it down, maybe I will build something else.
For many people 2020 has been a hard year, I am one of those.

Building a Twitter Clone

funkysi1701@gmail.com (funkysi1701) — Tue, 22 Dec 2020 20:00:45 +0000

I saw a tweet about building a twitter clone being harder than you would think. So this of course started me thinking how I would go about building something like that.

Ok so where would I start? First a few assumptions.

Development by a lone developer ie me
Tech stack will be dotnet and other tech I am familiar with
Database backend, probably SQL Server but I might use table storage for cost reasons should I try and actually build this. However if I design this well this should be something that could be swapped out as the system grows
User accounts on the system will be small as I can’t imagine anyone ever signing up. Why sign up to a social media platform with no users?

I guess the next question is what is Twitter?

A website that allows you to share 280 characters of text with your followers, allows you to follow other users updates and allows other user to follow your updates.

It also has an API that allow you to do almost everything that you can with the website.

Then there are of course mobile apps to consider but I am going to assume this is out of scope, however assuming a good enough API then this shouldn’t be a problem for future development.

First Steps

To start off with I would concentrate on the API, and then build a web client that makes use of the API.

So what would my MVP (minimum viable product) be?

User can authenticate with my API to get a token which allows access to other API endpoints
User can create a tweet
User can view own tweets
User can view tweets of another user
User can view tweets in their timeline
User can follow/unfollow other users
User can search for other users
User can search for keywords in tweets

I think that is probably sufficient to build my MVP for the API.

An interesting side note is that I could use the OAuth Twitter authentication to allow users to login to my twitter clone with the real twitter login details. However this makes no sense to me as we are essentially adding a dependency on the real twitter.

So what would I use for the frontend? I would start off with a Client Side Blazor frontend. Once I had a proof of concept that worked, I would think about styling and adding the UI elements that are familiar to twitter users.

The beauty of Client Side Blazor is that I can host cheaply in azure storage and distribute around the world via a CDN.

Due to the high number of times that follower and following count and other stats are queried I would consider storing these in the database and include a regular job to recalculate them so they don’t get out of sync with the data.

Having said all of this I am very tempted to fire up Visual Studio and see how far I get, and what problems I encounter along the way.

#005: Twitter Cards

funkysi1701@gmail.com (funkysi1701) — Sat, 12 Dec 2020 20:00:45 +0000

It has been a bit of a mad week this week. I joined a new team so lots of time learning what’s what and also being pulled in two directions as usual demands come through on top of that.

My blog runs on Blazor and I have been making use of JavaScript interop to update the html headers and update the page title to match the blog post article. This works great, I load the page and check the headers and they were saying what I wanted.

The problem was I wanted to add tags for twitter cards This means that when I paste a link to my blog on twitter you get a nice preview and pic of me in the tweet. This was not working at all even though I had the correct headers.

I eventually figured out that the problem was the fact I was using JavaScript to update my headers after the page had been initially loaded. Twitter was fetching my page before these headers got added and therefore couldn’t see the twitter card headers.

My solution was to use invalid html. Not ideal but it works. I added the required html tags in the body of my page using Blazor/C# instead of using JavaScript to add them into the header. Twitter appears to not be fussy in finding them in the wrong place.

Twitter provides a validator tool at Twitter Card Validator which my website now passes.

Not much else to say this week, apart from I am missing Visual Studio and C#, I have been mostly using VS Code on Linux and looking at php which isn’t as much fun as my usual day job.

Weekly Update #004

funkysi1701@gmail.com (funkysi1701) — Sun, 06 Dec 2020 20:00:45 +0000

I use sp_send_dbmail to send results of sql queries by email to business users. Recently an issue was raised that data was being cut off after 255 characters. To fix this I added @query_no_truncate = 1, however this stopped the column headings from being included. No idea why you can’t have all the data and column headings but there you have it.

What I am doing now is running 2 queries, one to get the headings, and one to get the data. In theory you should be able to combine them with a Union however you then have datatype issues for non text columns so I gave up with that idea.

My results have 60 something columns (don’t ask its for a data import into a third party system!) so I am not typing them all out. I can shove query results into a temporary table and then execute to get a list of columns.

SELECT name
FROM tempdb.sys.columns
WHERE object_id = object_id('tempdb..#TempTable')

However I need my list to be horizontal so I can use as column headers. I can use dynamic SQL and a pivot to flip them round.

DECLARE @cols AS NVARCHAR(MAX), @query AS NVARCHAR(MAX)

SELECT @cols = STUFF((SELECT ',' + QUOTENAME(name)
 FROM tempdb.sys.columns
 WHERE object_id = object_id('tempdb..#TempTable')
 FOR XML PATH(''), TYPE).value('.', 'NVARCHAR(MAX)'),1,1,'')

SET @query = N'SELECT ' + @cols + N' FROM
(
 SELECT name
 FROM tempdb.sys.columns
 WHERE object_id = object_id(''tempdb..#TempTable'')
) x
PIVOT
(
 MAX(name)
 FOR name IN (' + @cols + N')
) y'

Weekly update 003: progress notes

funkysi1701@gmail.com (funkysi1701) — Sat, 28 Nov 2020 00:00:00 +0000

Been a quiet week, so wasn’t expecting to have much to write on here, however a few things happened worth talking about.

My First PR

A comment was made to me to do something with the postcodes that are in the system I am developing. Find out what projects have postcodes near each other, and that way work can be grouped together and reduce potential mileage costs of staff that need to visit these projects.

A quick google search found https://postcodes.io/ which has an API that returns nearby postcodes. It also has a C# wrapper https://github.com/markembling/MarkEmbling.PostcodesIO

A comparison of what was being returned from the wrapper and what the API said should be returned revealed that the distance between postcodes wasn’t being returned.

As the code was on GitHub I could easily see how easy or difficult it might be to add the missing bit of information. It was easy! So, I forked the repo and made the change. I published the change to a private NuGet repo in my Azure Dev Ops account. That way I could try my revised package to check it did what I wanted.

I left a message on the GitHub project letting the owner know I had a potential fix for an issue. The project hadn’t been updated in over a year, so the owner may not be interested, or the project may have been abandoned.

I was in luck just 17 hours after I left a message the project owner said to create a Pull Request, which I did and shortly afterwards my code had been merged in and an updated version of the package existed in the public NuGet feed.

I have been thinking about contributing to open source for a while. However, I had not seen a project I wanted to contribute to, or a problem that I knew how to fix until now that is.

PHP

This week I had a call asking me if I knew PHP?

I did, over ten years ago, before I got my first IT job, I spent time learning PHP and MySQL. I created a blog, and I also created a website for my Dad’s camera club. The code I created back then was awful. No shared code, all the code was associated with the page, or sorts of bugs occurred and as time went by it became increasingly hard to update. The site was well liked but I eventually lost interest and moved on to learn other things.

This call led to me talking with the head of IT, and later a couple of the developers who have since granted me access to the codebase of a project.

I haven’t had time to spend a lot of time looking at the code so far, however this is nothing like the PHP I had built before.

The project makes use of the laravel framework and the first file I opened had methods and classes, so apart from the syntax you could think you were looking at C#.

Another thing that interested me was the project used docker containers, it has automated builds as well. Lots of modern programming ideas that I had some ideas about. I am looking forward to learning more about this project and how I might contribute to it.

Weekly Update #002

funkysi1701@gmail.com (funkysi1701) — Sat, 21 Nov 2020 00:00:00 +0000

I know Active Directory is fussy about clocks being in sync however not sure how todays issue happened.

I run my docker compose file from Visual Studio and I get a weird error.

SecurityTokenNotYetValidException: IDX10222: Lifetime validation failed. The token is not yet valid. ValidFrom: 'System.DateTime', Current time: 'System.DateTime'.

I deleted my containers, open and close Visual Studio a few times, nothing helps. Eventually I think to find out what the time is on my container. It has yesterday’s date. What has happened here? Surely recreating containers would have caused them to have todays date? I reboot and everything is fine again.

Turns out that it is a know issue, see https://thorsten-hans.com/docker-on-windows-fix-time-synchronization-issue I am using WSL2 and I have now changed back to using Hyper-V and the issue hasn’t come back.

Earlier in the week I spotted my build step was failing.

 - task: NuGetToolInstaller@0

Swapping to the next version of the step is all I needed to do to fix it.

 - task: NuGetToolInstaller@1

My guess is that support was dropped for this earlier version or there is some other incompatability with .Net 5.

Weekly Update #001

funkysi1701@gmail.com (funkysi1701) — Sat, 14 Nov 2020 00:00:00 +0000

One of my favourite podcasts is Troy Hunts weekly update. In it he discusses stuff that he has been working on, plus some personal stuff. I am going to attempt to do something similar. It will probably take me a few of these before we get a look and feel that works.

Monday

A week off work, mainly to use it up before year end, plus want to get a few jobs around the house done.

I did ask the following question on Twitter.

Hey #azurefamily and #dotnet developers how do I get more involved in mentoring?
— Simon Foster (@funkysi1701) November 9, 2020

As a one person dev team, my biggest weakness is working with others so any ideas of how to change that are great.

Tuesday

Dotnet 5 is out! The latest version of dotnet is released by Microsoft and to celebrate there is dotnetconf to listen to. Due to time zones and family commitments, I haven’t listened to an awful lot of it but I did see the keynote and loved the 3 Scott’s chat.

Wednesday

My youngest son was 3 today, due to Coronavirus we didn’t do much but we celebrated as a family, and he even had a zoom call.

Thursday

Blazor has a new feature Virtualize where a list of items can only load the ones on screen. I have been trying to get this to work on my blog, works great running locally but not working in production yet.

Think I know what might be happening. I use Cloudflare to do my SSL, as Custom SSL certs for the cheaper Azure Web Apps is not supported. Something in Cloudflare is caching or interfering.

https://zimmergren.net/solved-asp-net-core-blazor-web-sites-does-not-work-with-cloudflare-html-minification/

Turning off HTML minification fixed my issue!

One additional thing I added to my Blog is the config page which details some of the config settings. I think this probably came from https://www.hanselman.com/blog/adding-a-git-commit-hash-and-azure-devops-build-number-and-build-id-to-an-aspnet-website but it was a while ago when I first did this on another project.

At the moment we have .net Version, Commit and Build links.

The .Net Version is obtained from

@System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription

A few other bits of info can be obtained from System.Runtime.InteropServices.RuntimeInformation which I have included on the page for fun. There are probably security concerns with exposing all this info publicly so something to bear in mind if you try this.

Build Info is passed to my code by a build step

- script: '(echo $(Build.BuildNumber) && echo $(Build.BuildId)) > .buildinfo.json'
 displayName: "Emit build number"
 workingDirectory: '$(Build.SourcesDirectory)/src/WebBlog'
 failOnStderr: true

This simply passed the build id and number which are stored as variabled and saves them in a text file.

I then have a class that reads them and constructs a link.

using Microsoft.Extensions.Hosting;
using System;
using System.IO;
using System.Linq;
using System.Reflection;

namespace WebBlog
{
 public class AppVersionInfo
 {
 private const string _buildFileName = ".buildinfo.json";
 private readonly string _buildFilePath;
 private string _buildNumber = string.Empty;
 private string _buildId = string.Empty;
 private string _gitHash = string.Empty;
 private string _gitShortHash = string.Empty;

 public AppVersionInfo(IHostEnvironment hostEnvironment)
 {
 _buildFilePath = Path.Combine(hostEnvironment.ContentRootPath, _buildFileName);
 }

 public string BuildNumber
 {
 get
 {
 if (string.IsNullOrEmpty(_buildNumber))
 {
 if (File.Exists(_buildFilePath))
 {
 var fileContents = File.ReadLines(_buildFilePath).ToList();

 if (fileContents.Count > 0)
 {
 _buildNumber = fileContents[0];
 }
 if (fileContents.Count > 1)
 {
 _buildId = fileContents[1];
 }
 }

 if (string.IsNullOrEmpty(_buildNumber))
 {
 _buildNumber = DateTime.UtcNow.ToString("yyyyMMdd") + ".0";
 }

 if (string.IsNullOrEmpty(_buildId))
 {
 _buildId = "123456";
 }
 }

 return _buildNumber;
 }
 }

 public string BuildId
 {
 get
 {
 if (string.IsNullOrEmpty(_buildId))
 {
 var _ = BuildNumber;
 }

 return _buildId;
 }
 }

 public string GitHash
 {
 get
 {
 if (string.IsNullOrEmpty(_gitHash))
 {
 var version = "1.0.0+LOCALBUILD";
 var appAssembly = typeof(AppVersionInfo).Assembly;
 var infoVerAttr = (AssemblyInformationalVersionAttribute)appAssembly
 .GetCustomAttributes(typeof(AssemblyInformationalVersionAttribute)).FirstOrDefault();

 if (infoVerAttr != null && infoVerAttr.InformationalVersion.Length > 6)
 {
 version = infoVerAttr.InformationalVersion;
 }
 _gitHash = version[(version.IndexOf('+') + 1)..];
 }

 return _gitHash;
 }
 }

 public string ShortGitHash
 {
 get
 {
 if (string.IsNullOrEmpty(_gitShortHash))
 {
 _gitShortHash = GitHash.Substring(GitHash.Length - 6, 6);
 }
 return _gitShortHash;
 }
 }
 }
}

The BuildId and BuildNumber properties just fetch the details saved into the text file during the build. This can then be passed to build the build link.

<a href="https://dev.azure.com/{OrgName}/{RepoName}/_build/results?buildId=@appInfo.BuildId&view=results">
 @appInfo.BuildNumber
</a>

Finally, the GitHash properties need to fetch the hash and shorthash of the commit which is a bit more complex. This is achieved using the following line in your build.

- task: DotNetCoreCLI@2
 displayName: 'Publish'
 inputs:
 command: 'publish'
 publishWebProjects: true
 arguments: '--output $(Build.ArtifactStagingDirectory) /p:SourceRevisionId=$(Build.SourceVersion)'

/p:SourceRevisionId=$(Build.SourceVersion) add the revision hash to [assembly: AssemblyInformationalVersion] during the build which can then be extracted using the gitHash property above, before being passed into the commit link.

<a href="https://github.com/{OrgName}/{RepoName}/commit/@appInfo.GitHash">
 @appInfo.ShortGitHash
</a>

Security Headers

funkysi1701@gmail.com (funkysi1701) — Sat, 26 Sep 2020 00:00:00 +0000

Have you wondered what info you are leaking via your response headers?, do you want some kind of guide about what headers to set or remove altogether?

Head on over to https://securityheaders.com/ This is a site created by security expert Scott Helme that rates a URL based on what response headers it can see.

I am pleased to say https://www.funkysi1701.com is now getting an A.

So how do you add/remove headers in dotnet core?

In my configure method in Startup.cs I have the following code block.

app.Use(
 next =>
 {
 return async context =>
 {
 context.Response.OnStarting(
 () =>
 {
 context.Response.Headers.Add("Permissions-Policy", "microphone=()");
 context.Response.Headers.Remove("Server");
 context.Response.Headers.Remove("X-Powered-By");
 context.Response.Headers.Remove("X-AspNet-Version");
 return Task.CompletedTask;
 });

 await next(context);
 };
 });

I have only included a few of the headers I am adding as the excellent https://securityheaders.com/ can tell you which headers you should add and what options you might want.

Returning to blogging as a developer

funkysi1701@gmail.com (funkysi1701) — Fri, 25 Sep 2020 20:00:45 +0000

My last blog post was over six months ago.

Covid 19 has hit the world, and I will be honest I have found it a challenging time.

My Blog had gotten into a bit of a mess. It had become fragmented with different versions of the same thing; I will attempt to explain what has become of my blog.

The original WordPress site can currently be found at https://www.pwnedpass.com/ I would prefer it to be on a sub-domain of funkysi1701.com but for some reason I haven’t been able to get that to work, not sure if it is a limitation of my hosting package. I like WordPress, it is very flexible, easy to get blog posts out there. But I want to write content about development and having a site I can tinker with is important to me.

Most of my WordPress blogs have been imported into dev.to and a few extra have been written on this platform. I like dev.to it is a wonderful place to share content and it has one or two extra features I like.

dev.to has an integration with Stackbit/Netlify and this became https://www.funkysi1701.com . I like having a personal site, but having the same content as dev.to. To add content to this site all I need to do is write it on dev.to and some magic will go on behind the scenes and new content will be published.

However, as a developer I don’t like magic, I want to understand what is going on a fiddle with all the settings and make it do what I want.

dev.to has an API, I can build a site in .Net Core and make API calls to fetch the content I want. I understand APIs, I understand .Net and can customise my site exactly how I want it, plus play about with a .net website. This is what https://www.funkysi1701.com is now.

So what have I built so far. I have a Server Side Blazor site running .Net 5. Why Server Side and not Client Side I hear you ask? Well only because I have more experience with Server Side and know how to quickly create a website with that technology, I may change it as time goes by, but we will see.

I have two pages a list of my blog posts and a page that displays the content. Both of these use the dev.to API. I lied, there is a third page I hacked together to do some page redirection from the WordPress URLs. This is something I will change as time goes on.

There are lots of improvements I want to do, there are probably also lots of broken images or links as well. Hopefully, this will result in a good platform to blog about as well as on.

Testing expiring SSL certificates

funkysi1701@gmail.com (funkysi1701) — Tue, 03 Mar 2020 20:00:45 +0000

Let’s Encrypt is amazing, you can easily add SSL certificates to any website and automate the renewal process. I have talked before about how impressive it is.

Once you start adding SSL certificates to your production sites however you may want to check when they expire so you don’t get caught out. You can always open your site in your favourite browser and view the certificate information and expiry date.

However there is a way to automate this check.

[Fact]
public void IsSSLExpiring()
{
 var handler = new HttpClientHandler
 {
 ServerCertificateCustomValidationCallback = CustomCallback
 };
 var client = new HttpClient(handler);

 HttpResponseMessage response = client.GetAsync("https://www.example.com").GetAwaiter().GetResult();
 Assert.True(response.IsSuccessStatusCode);
}

private bool CustomCallback(HttpRequestMessage arg1, X509Certificate2 arg2, X509Chain arg3, SslPolicyErrors arg4)
{
 var now = DateTime.UtcNow;
 var expire = arg2.NotAfter;
 var diff = (expire - now).TotalDays;

 Assert.InRange(diff, 30, 1000);
 return arg4 == SslPolicyErrors.None;
}

This code gets the SSL expiry date from https://www.example.com and will fail the xunit test if the expiry date is less than 30 days in the future. I then schedule my tests to run regularly on all my environments with a Let’s Encrypt Certificate and this gives me advanced warning if a SSL certificate is about to expire.

The Assert.InRange(diff, 30, 1000) line will fail the test if the expiry date is less than 30 days or greater than 1000, but as the default expiry for Let’s Encrypt certificates is three months it will never be greater than 1000 days even with a freshly installed certificate. These values can be tweaked to suit your use case, however 30 days is enough time for me to investigate what is happening.

To execute my tests I use a scheduled build in Azure DevOps, but anything that regularly can run your tests will do the job.

The code above is just a simple example to get your started for my purposes I have put all my URLs into config files and just pass these into my tests, so I don’t need a custom test for every different URL.