2016 on Funky Si's Blog (Dev)

What should be in Source Control?

funkysi1701@gmail.com (funkysi1701) — Thu, 24 Nov 2016 20:00:45 +0000

I am currently working on source code that is over 5Gb in size. This is mostly due to a poorly thought out folder structure, there are code files, images and Excel files all jumbled together. I think a clear distinction should be made between source code and data.

Source Code

I will define source code as anything that is written in order to compile and run the project. If it is a webpage it will be all the HTML, CSS and Javascript or any file used to produce these. I would also include any configuration files and files used to build/deploy the website or project. Anything that is compiled from your source files can safely be ignored.

Data

I would define data as anything that is added to the project during its life. So if you have an upload option, anything that is uploaded I would describe as data. The site should still function without (or very little) data.

Images

Images can fit into both groups. Any icons or images attached to the functionality of the project I would class as source code. However anything that is uploaded should be classed as Data.

Databases

The database should also be classed as both. The data, anything that is inside a database table should normally be classed as data. Stored Procedures, Functions and Views are all Source Code and would benefit from version control.

Source Control != Backup

Source control is not an excuse not to backup things. Don’t just commit files to source control so you know you can restore them if you need to. Files in general in source control are there so you can see how they changed over time as the code base changed. Files in you backup are a snapshot of what the application was at a point in time and will include ALL the data.

One last point before I end. If you are hosting on a Cloud Computing platform like Azure it gives you an easy way to distinguish between Data and Code.

Anything in your

Web App = Code
Blob Storage = Data
SQL = Data/Code

Each project is unique and there will always be exceptions to these suggestions but I think this is a good goal to have. What do you think?

Getting control of a codebase

funkysi1701@gmail.com (funkysi1701) — Thu, 17 Nov 2016 20:00:45 +0000

So recently I started working on a new codebase. I will be honest when I first saw it, it was a mess. Here are a few of the things I did to try and regain control.

I was given access to the source code on Visual Studio Team Services. However this consisted of a single commit 3 months ago. When I looked at what was running on the production server it was clear that changes were being made live with no regard for source control.

The first thing I did was commit everything that was running live into source control.

Next I created a SQL Server Data Tools (SSDT) project to keep track of all the database objects. Previously there was a folder with some stored procedures in it, but these did not match with what was currently running.

I now had in source control the current state of the website and the database, so I knew I could get things back to this state if I made some bad changes.

Lets start by looking at the website code I had. There was no solution file, the only way to look at the website was to setup my local IIS to run what was in the website folder. I could then use Visual Studio to “open” my local IIS website and attach to process to debug it.

Next I Looked at Default.aspx to see how the website worked. The majority of the website code was stored in the database stored procedures. After the tag there was a <% %> which contained a Response.Write(RunSP.RunStoredProcedure(Parameter1, Parameter2, …) command, which executed a stored procedure and the results of the stored procedure was the html code including any javascript that the webpage needed to display. I will be honest I have never seen any code like it. My guess is that the developer was secretly a DBA and wanted to make any web page changes by just changing how the stored procedures work.

This meant that the website is not going to do anything without a backup of the database running, and meant my SSDT project was going to be vital. However the database was in a bad state, it consisted of a fair few broken objects and SSDT would not build.

Using find I went through each of the broken database objects to find where in the code they were being used. Luckily most were referenced in commented out code, so I just removed all the broken database objects. The database could now be built. However there was a dependency on the users table of another database. (This was the developers solution to sharing logins between websites) As I was using SSDT I added a database dependency, problem solved for now.

Next I tried publishing my database. SQL CMD encountered a parsing error. The reason for this was my SPs contained javascript eg $(document), SQL CMD uses $(DatabaseName) as variables for different database so it was getting itself confused.

My solution was to use Find and Replace to replace all the $ with ’ + CHAR(36) + '

So I now have a SSDT project that builds and publishes but still no website project.

To get the website running from Visual Studio I started off creating a .Net 4 website project and added Entity Framework 5 and MVC 3 via nuget. I then copied all the website code into the new project, and with a bit of work I got it to build. Most of the work was relating to namespaces and referencing the correct one and moving the EF model from AppCode to a custom named folder. A bit of trial and error later I had a version of the website that could be run from Visual Studio.

I have not deployed my new version of the website as it needs further testing. No automated testing or even a smoke test checklist currently exist.

As my source code is hosted on Visual Studio Team Services (VSTS), I can get VSTS to build each commit and check I haven’t broken the build. This is not that helpful at the moment, hopefully one day I will have automated tests that can be run here as well.

Wow, I feel like I have done loads with this code so far but there is loads more still to do. I need to understand more about the business processes behind the code with a hope to understand why some architectural decisions have been made. I want to refactor the code as much as is possible, I would like to remove much of the html/javascript from the stored procedures as I can’t see that there is any advantage to running a website like this. Please correct my if I am wrong.

LINQ

funkysi1701@gmail.com (funkysi1701) — Thu, 06 Oct 2016 20:00:45 +0000

What is LINQ?

LINQ is an acronym for Language Integrated Query, which describes where it’s used and what it does. The Language Integrated part means that LINQ is part of programming language syntax. In particular, both C# and VB are languages that ship with .NET and have LINQ capabilities.

How do I use LINQ in my C# code?

To use LINQ the first thing you need to do is add the LINQ using statement.

using System.Linq;

In your code you need a datasource, for this example I am going to use a simple array, but it can be anything eg SQL, XML etc

int[] data = new int[10] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };

Next you need a LINQ query. (Note I know the Q in LINQ means query, so I have just written query query, if you are one of those people who hates seeing PIN number you might not like this blog post.) A LINQ query is very similar to a T-SQL query, so if like me you are good with databases this should make sense to you.

In T-SQL you can have:

SELECT num
FROM data
WHERE num = 1

In LINQ this becomes:

var query =
from num in data
where num == 1
select num;

Finally you need to do something with the query you have written. I am just going to print the results of my query to console.

foreach (var num in query)
{
 Console.Write(num);
}

What other SQL like syntax can I use?

In T-SQL you can control ordering using ORDER BY, LINQ has a similar syntax orderby

orderby num descending

In T-SQL you can use GROUP BY, to do something similar with LINQ

group num by num.Type into type
select type

foreach (var type in query)
{
 Console.Write(type.Key);
 foreach (var num in type)
 {
 Console.Write(num);
 }
}

JOINS

So you thought joining tables was a SQL Server only thing. Think again you can do this in LINQ

var joinquery =
from cust in customers
join prod in products on prod.CustomerId equals cust.Id
select new { ProductName = prod.Name, CustomerName = cust.CompanyName };

Conclusion

There are loads more LINQ functionality that you can use. While writing this blog I found https://learn.microsoft.com/101-LINQ-Samples-3fb9811b which has loads of examples of different queries that you can write with LINQ.

This has inspired me to use LINQ more in my code and learn more about the different queries that could be written.

Looking back Ten years

funkysi1701@gmail.com (funkysi1701) — Thu, 29 Sep 2016 20:00:45 +0000

Next month I will celebrate ten years working at my current job, two weeks after that I will start a new chapter of my life at a new company. Lets take this opportunity to look back ten years at some of the great stuff I have achieved.

2006

In 2006 I had no IT experience. If you were to ask me where Event Logs could be found I wouldn’t know, and the letters DNS meant nothing to me. I was quiet and shy but hard working. The idea of making IT my career hadn’t occurred to me, this was just a job.

I started off life doing first line support and compiling health and safety files onto CD-ROMs (known by the company as Eurofiles) with HTML. At the time the company had a mix of Windows 2000 and Windows XP, using MS Office 2003. Server wise I can’t remember exactly but I think two domain controllers, a database server and a backup server using a tape drive. At this point in time I don’t believe I did a lot with the servers but I think the servers were running Windows Server 2005.

As time went on I learnt more and more about what the company did and how stuff worked.

Nagios

One of my early achievements was setting up a nagios server. I have blogged about Nagios before, but it is a server monitoring system that runs on linux. I am extremely proud of what I have achieved here, no one else in the company had my knowledge of how nagios worked. Initially I even experimented with using a cheap mobile phone connected to the serial port (remember when PCs had these?) to send SMS messages to alert of down time. However this was abandoned when I burned through all the credit on this phone.

Nagios is still used today, and my current team have been shown how to extend the system as the business changes. My philosophy has always been if we have a problem that Nagios didn’t warn us about we need an extra nagios test. These days the nagios web interface is publicly available on the internet so can be checked from anywhere (assuming the office internet is up!) and a mobile app replaces the SMS idea we originally had.

2011

At the end of 2010 the IT Manager resigned, then the Developer resigned and finally the Lead Developer resigned. I was alone. I was IT Manager. I negotiated my first pay rise since joining the company. I was IT.

Wow things were stressful back then. I had so much to learn but somehow I managed. I learnt how to Interview, yes I have employed some clangers along the way, but also some great staff. As well as internal staff I learnt to deal with contractors. We had contractors to help with our internal systems and also contractors to do development work. One thing I have learnt about contractors is that you will always have to chase them at some point to deliver what you are paying them for.

Server Migration

Around the start of 2011 our email server started to show its age. Exchange 2003 had a hard limit on the size of its information store and we were rapidly approaching this. Along with an IT contractor I worked to migrate to Exchange 2010, this was a huge project and caused all sorts of issues which we just worked through. Since then I have done other migrations so Windows Server 2012 and also virtualized many servers in 2013.

Bandwidth

A major limit with our infrastructure has always been the internet connection coming in. For a while we tried to load balance three ADSL connections, but the upload speed was always a limiting factor. It was a major victory for the simplification of our network when we got a leased line installed into our head office, helped by a government grant we gave us free installation. Not content with this I did the same again for our second York office, including the free installation.

Learning Development

Since I started I have always been learning development stuff. But in the past few years I have learnt a massive amount helped largely by two main things. My boss passing lots of my responsibility onto others and dedicating lots of my time to development tasks, and also the opportunity to learn with our outsourced development team.

Finishing off

There are a few things that have been ongoing that we have wanted to change since I started which I can’t really take the credit for as they are not complete yet, but I am proud as they may be finished in the next few months.

The company stores calendar information in one giant excel spreadsheet, this is being replaced by outlook calendars.

Our last Windows Server 2003 server and tape drive is being decommissioned. This is the only server that has been running for the entire time I have been here.

The way the company keeps track of work coming in and who does what is being reviewed. Over the years many people have tried to adapt the old database that one of the directors created in Access many years ago. I did a major overhaul recently to delete unused columns and added extra invoicing functionality. What is needed is a fresh system, maybe a CRM can do everything they want. I wish them lots of luck in doing this as I know it won’t be easy but it has to be done.

2016

There are many other projects and pieces of work that I have done that I am proud of. In 2016 I will leave a stable IT department. All clients are on Windows 10 and using Office 2010. We have virtual servers running Server 2012 and are using some great services from Azure to run our SaaS websites. There are of course things I would have liked to achieve, and things I wish I had done differently but on the whole it feels like the right time to move on, especially now long running projects are starting to conclude.

10 Ways to Survive as an IT Manager

funkysi1701@gmail.com (funkysi1701) — Thu, 22 Sep 2016 20:00:45 +0000

So after five and a bit years of being an IT Manager here is some advice I have learned along the way in no particular order. On the whole I have enjoyed myself but it has been a real challenge at times.

1. Figure out what plates are still spinning

Being an IT Manager is all about keeping everything running all of the time. A bit like spinning 5 or 6 plates. You have plates for your servers and network infrastructure, you have plates for bespoke databases that you maintain, you have plates for your staff (including any external contractors), you have plates for any websites or apps that you develop. That is a lot of plates to keep spinning and that before you start thinking about what your boss wants you to deliver. Make sure you know what is happening with all these plates, which ones are happy, which ones are on the way to the floor and which ones you need to get the glue out and repair.

2. Make it someone else’s problem

If you can blame someone else do so. If your internet goes down it is your ISPs fault. If your website dies its your hosting company’s fault. Take responsibility for problems but if when something goes wrong you can pick up the phone and ask for help, it will make your life easier.

3. Hire good staff

Hiring poor staff wastes time and money and makes you look bad by others. Demand the highest salary band for new staff that you can afford and don’t agree to hiring anyone that you have doubts about. It is easy to bow to the pressure to get someone quickly but this will always result in worse problems in the long run. Once you have a good team do your best to keep them, and warn upper management of the problems if staff leaves (basically make it their problem not yours!).

4. Learn, Learn, Learn

You may or may not have the opportunity to go on training courses. Whatever your situation spend time learning new stuff that will benefit the company and yourself. You can learn a lot by reading online, you can petition for training from your managers, you can fund training yourself, you can ask for help from your different suppliers. The more you learn, the more you can do and the more useful you can be to the company, plus the more interesting you will find the job.

5. Say No!

Don’t be afraid to say no. You will always be asked to do the impossible and if something is impossible say so at the start. It wastes everyone’s time if you spend a lot of time trying to do the impossible. Always give your reasons for saying no, and if you always say no people will think you are unhelpful. A better way to say no is to come up with a better solution. No I can’t do it your way but here is a better solution.

6. Don’t give estimates

If you are asked how long something will take you don’t answer straight away or give an exaggerated estimate. Go away and spend some time thinking of everything that is involved before replying. There will always be something that you forgot to consider when first asked about it and looking at the different components will help plan out the work needed as well as provide an estimate.

7. Know what to tell your boss, and what not to

This is a hard one to get the balance right for. You need to tell your boss enough so that they appreciate all that you do, but too much and they will stop listening and accuse you of talking in technobabble. I have never got the balance right with this one. I have always aired on the side of not telling my boss enough, and hence they don’t realize that I saved the day on Sunday night as everything is working again on Monday. Do repeat yourself. If your server is running low on resources start asking for replacement hardware early, and increase the frequency and the panic in line with the problems it is causing.

8. Understand the problems of the business

Businesses need to make money. If the one you work for isn’t making enough money you will soon be looking for another. If you work for IT you will quickly start to see the problems of the business, think about what simple changes IT could make to improve things that would benefit the whole company. Some of your suggestions won’t go anywhere, but some may have a massive impact. I can think of a few changes that IT have spearheaded that I am very proud of, upgrading our internet connection, simplifying or automating processes and delivering new versions of software.

9. Ask for help

Don’t be afraid of asking for help. There are lots of places to look for help. Other departments could take more on, you could recruit extra help, you could hire external contractors. You can ask questions on support forums like ServerFault or StackOverflow , many software re-sellers or other suppliers are a good point of contact for questions about things they supply. Microsoft Support was also invaluable for a server issue.

10. Think about disasters

Write a disaster recovery plan or backup policy. Yes there will always be something more important that needs doing, but just stop for a moment to think how you would feel if everything died on your watch. The one thing you can rely on with technology is that it will fail at some point. A back of the envelope plan of action is better than no plan at all, even better is a detailed plan of what to do when each and every service you rely on fails. Plan additional services with an idea of adding extra redundancy. Always have multiple Domain Controllers, think about what data you could run from the Cloud. VMs could be replicated to the Cloud, and servers could be run from there.

Imposter Syndrome

funkysi1701@gmail.com (funkysi1701) — Thu, 15 Sep 2016 20:00:45 +0000

This week I handed in my notice at a job I have had for almost ten years. In a few weeks time I will start a brand new job as a web applications developer.

This is a great achievement for me and a great chance to learn and expand my development skills.

However moments after I handed my notice in, a huge wave of uncertainty enveloped me. I have read about this and I think I was suffering from an attack of Imposter Syndrome.

Despite external evidence of their competence, those with the syndrome remain convinced that they are frauds and do not deserve the success they have achieved. Proof of success is dismissed as luck, timing, or as a result of deceiving others into thinking they are more intelligent and competent than they believe themselves to be.

I started to convince myself I couldn’t do the job I had just accepted and that moments after I arrived on my first day I would be found out as a fraud and kicked out the door.

My logical brain was no match for the Imposter Syndrome. Having been kicked out the door of this new job, I would become penniless in no time with no employer (even my previous boss wouldn’t want me back) or recruitment agency willing to speak with me.

OK lets see if we can fight off the Imposter Syndrome and look at a possible worse case scenario. I start at my new job, it is very hard work and a month or so later I have to move on.

No employer ever would sack me on my first day. My first day will be lots of learning how they do things, looking at procedures. I have lots of experience of learning new things, looking at procedures and if I put the work in I can make this new job a success. I have lots of good ideas and useful experience that no one at my new company has.

I don’t know what the solution is to Imposter Syndrome other than to try and ignore it by thinking positively and the knowledge that I have done amazing things in the last ten years and I will do the same in the next ten.

DZone

funkysi1701@gmail.com (funkysi1701) — Thu, 25 Aug 2016 20:00:45 +0000

For a while now I have been sharing some of my blog posts on the Dzone website.

The Dzone website allows users to submit links to content and I have been submitting the content I have created on this website. This is how Dzone describes themselves:

With over 1 Million members, DZone.com is one of the web’s largest communities and publishers of technical content for software professionals. Developers from all over the world come to DZone for the latest and best content to hone their skills and advance their careers.

Well this week I have been invited to join the MVB (Most Valuable Blogger) programme. The hope is that more of my readers will find my content from the DZone website.

The DZone team will soon start sharing my content on the DZone website and I hope this will result in even more people reading what I have to say.

If nothing else this is clearly telling me that my blog is making a difference, people are taking note of me and what I have to say. I need to keep going and keep writing articles and be more consistent.

Amazon Web Services Pt 2

funkysi1701@gmail.com (funkysi1701) — Thu, 04 Aug 2016 20:00:45 +0000

Last time I started looking at Amazon Web Services and how it differed from Azure. I am going to continue looking at what it can do.

Virtual Machines

Lets look at what you can do with Virtual Machines. I selected to create a new Virtual Machine (or as AWS calls them an EC2 Instance)

First you choose a name for your VM and then the OS that runs on it. There are 5 main OSes to choose from Windows Server, Amazon Linux and a selection of the most common Linux flavours.

You can then download a certificate to secure your VM.

Like Azure, AWS takes a few moments to create your VM. While I wait I can see that AWS has configured a firewall so only my current IP can connect to it.

Once the VM is ready you can download an RDP file. However to get the login details you need to decrypt the password using the certificate you downloaded when you created the VM.

It is interesting to compare the difference in security between Azure and AWS. Azure allows the resetting of passwords of VMs directly from its console, however I suspect that in AWS if you loose your certificate (AWS states they don’t keep a copy of this) you would have to recreate your VM.

Like with the Websites the default name of the VM is much less user-friendly than what you get from Azure. However I suspect there are other options I haven’t spotted that may customise these.

Azure Portal vs AWS Console

I really like the Azure Portal. It feels like something that has been designed so you can easily access all the options for a specific Azure feature.

The AWS Console probably has all the same options as with Azure however I don’t think it looks half as good, and will take me a while looking through menus to find the equivalent options. Part of this is due to my unfamiliarity with AWS, so will get easier with time.

Amazon Web Services

funkysi1701@gmail.com (funkysi1701) — Thu, 21 Jul 2016 20:00:45 +0000

I am a big fan of Azure but I know zero about its biggest rival – Amazon Web Services or AWS.

So lets sign up for a free trial and see what it can do.

The AWS free trial is available from https://aws.amazon.com/free/ and lasts for 12 months. From memory I think the Azure free trial lasted only one month.

To start you need to login with your amazon account and create an AWS account. This requires your name and address and your payment info (you will only get billed if use services not covered by your free trial).

Interestingly AWS requires you to verify your identity via an automated phone call. (I don’t recall doing anything like this for Azure but please correct me if I am wrong.)

Once you are logged in you get a series of links displaying all the different services that are available. First impression is this is a simpler view to Azure’s portal with a similar amount of services. At the top right is an option to select which region you want to use, in Azure I use North Europe and West Europe, AWS has Ireland and Frankfurt.

Create a Web App

First thing to try is setting up a website. I selected create a web app and I get a page asking me for its basic details (very similar to Azure, however AWS asked what language your code is written in, Azure handles all of these) AWS websites appear to support a host of different options similar to Azure.

The actual creation of your website takes a few moments (like on Azure). However the default URL for websites is similar to https://test.vjbbimyv7w.eu-central-1.elasticbeanstalk.com/ which is not quite as nice as the Azure equivalent https://test.azurewebsites.net

Azure has a host of command lines available via powershell. AWS has a similar command line interface called AWS CLI, including the option to deploy from git to your website.

AWS Toolkit for Visual Studio is an extension that allows for the publishing of websites to AWS. (Just like you can publish to Azure)

As I learn more about AWS I will continue to blog about it. Amazon Web Services Pt 2

Clever things with MS Access

funkysi1701@gmail.com (funkysi1701) — Thu, 07 Jul 2016 20:00:45 +0000

I hate MS Access and especially developing with it as you can’t do any thing clever with it.

Wrong, Wrong, Wrong!

There are a few clever things I have been able to script to make developing with it passable. I still would rather use Visual Studio but this improves the experience a fair bit.

Source Control

Access files are binary (I use the ADP/ADE file format but I assume other Access file formats have the same problem) and so you can’t diff them to see what has changed. This is bad.

However there is a solution to this. A tool called Access SVN and can be downloaded from https://accesssvn.codeplex.com/ , this gives you a way to extract to text files all the forms and reports that are in Access. Before every commit I would manually run this tool on my ADP file and extract to text files, then I would commit these text files to source control and could easily see what had changed in each commit.

Despite the name Access SVN, the tool is not tied to subversion, you can use any source control system, I use git.

Also included in this tool is a way to do this with the command line, so you can make this a build step on your build server. I have not used this extensively yet, but the syntax is fairly simple

asvn.exe e "path to Access file" "path to txt files" "*.*"

The filter at the end . allows you to specify what to extract so you could extract all forms/reports beginning with D with “.D”. I had trouble using . because the names of my forms/reports contain characters not allowed in a windows file name. I am sure there is a way round this but I haven’t had chance to look into it further yet.

Testing MS Access

Surely testing is not possible with MS Access? I would have agreed with that statement the other day until I found a neat way of testing if a feature is enabled.

Firstly a bit of background. I develop using MS Access 2003 because the design view is far easier to use, however because it is out of support all my users use MS Access 2010. MS Access 2010 has a feature called Tabbed Documents which allows all forms and reports to open in new tabs so you can easily switch between them. This feature can only be enabled in MS Access 2010 and has no effect if opening with MS Access 2003.

If you use Access SVN on your Access file with tabbed documents turned on and off you will see UseMDIMode: 0 and UseMDIMode: 1 show up in the Database properties file. UseMDIMode: 0 means that tabbed documents is turned on.

In powershell I can now write a test to see if UseMDIMode: 0 can be found in the database properties file

Get-Content "General\Database properties.dbp.txt" | Select-String "UseMDIMode: 0" -quiet

If the test passes True will be returned, if it fails null will be returned.

On my build server I scripted the extraction of Database properties.dbp.txt from the ADP file with asvn.exe before running this test. While not strictly needed as Database properties.dbp.txt should be in source control, it is possible that someone could forget to extract the text files from the ADP, with this step you are always testing what is enabled in the binary file.

MS Access Connection Strings

While developing with MS Access I often swap the database connection to point to my local machine or a build server. I always try and remember to only ever commit with this set to the live database to avoid obvious problems.

The other day I found on stackoverflow a way to script this. I love this! I can include this step in my deployment process and it will overwrite what ever the connection string is in source control with what your production environment needs.

All you need to run this step is, (note it is spaces between the parameters not commas)

cscript connect.vbs Project.adp "ServerName" "DatabaseName"

The contents of connect.vbs can be found on the stackoverflow article . It is also possible to pass username and password if your environment requires this.

Compiled ADE

The last clever thing I do with MS Access is convert my ADP file into the compiled ADE version. To manually do this there is an option in the tools menu.

To automate this I run

cscript createADE.vbs "path to ADP" "path to ADE"

The contents of createADE came from this forum post , the only change I made was to comment out some of the echo statements so it would run silently as part of my build process. It should be noted that cscript and wscript are almost identical and either will run these scripts however in a command line environment cscript is preferable, and wscript should be used in a windows environment.

I am quite surprised at how much I have managed to do in terms of scripting the build and deployment process for MS Access. I still don’t like developing with Access but this has definitely improved things.

Periodic Table of DevOps

funkysi1701@gmail.com (funkysi1701) — Thu, 30 Jun 2016 20:00:45 +0000

The periodic table lists all the chemical elements and groups them together based on some key properties. Today I found an article about the periodic table of DevOps .

I am not going to discuss every element but I thought I might go through some that I have heard of or used.

Github – The repository of lots of open source software. My Github can be found at https://github.com/funkysi1701
Amazon Web Services – The second most popular cloud computing provider (not sure why this isn’t number 3?)
Git – The distributed source control system that everyone uses these days.
Azure – The number one cloud provider, I have used this a lot, mostly with websites but also with some of their other features like Traffic Manager.
Bitbucket – like Github but allows private repositories. I have used this extensively for work based projects that I don’t want to be public.
Google Cloud Platform – don’t know much about this one, but no surprise that google wants a piece of the cloud computing pie.
Selenium – This is a product I want to play about with as allows front end testing with a browser.
Rackspace – Before we made the jump to Azure we made use of some Rackspace servers.
Subversion – The first source control system that I used, but been using git so long now not sure I can remember how it worked.
Visual Studio – The IDE from Microsoft that I use to write code. I am a big fan as it does everything I could want.
TeamCity – The continuous integration software that I have been using to automate my deployments.
MSBuild – This is used by Visual Studio to build your software and can also be used by your deployment scripts.
Trello – A website that allows you to create a board of ideas or things to do.
Slack – Brings all your communication together in one place. It’s real-time messaging, archiving and search for modern teams.
New Relic – A software analytics tool suite used by developers, ops, and software companies to understand how your applications are performing. Useful but find myself favouring Application Insights (part of Azure) more now.
Nagios – Yay nagios is on the list! My favourite server monitoring system.
Splunk – This application can be used to search, monitor and analyse all your log files to find out what is happening. Don’t currently use it but I have tried it out in the past.

What is your favourite DevOps tool? Why not leave a comment below?

Automatic Git Tagging

funkysi1701@gmail.com (funkysi1701) — Thu, 16 Jun 2016 20:00:45 +0000

One of the features of git is the ability to tag a point in my change history with a tag. For a while now I have been manually tagging my code whenever I do a release, so I can easily work out what has changed by doing a diff between two tags.

Now that I am automating my release process with TeamCity I am thinking about how to manage my tags better.

TeamCity has a setting called VCS Labeling which comes in very handy.

Configuring it is fairly simple as it only has three settings.

VCS root to label: This is obviously the url to your git repository Labeling pattern: This is the text of the label to be added. Label successful builds only: Do you really want to add a tag if the build failed?

A tag needs to have a unique name, so adding a tag just called deployed won’t work. When I used to add tags manually I used the naming convention of deployedyyyymmdd. While this naming convention is possible with TeamCity I use something a bit more complex to provide more information about what has been deployed.

TeamCity provides lots of parameters that can be used in your build steps and also in the Labeling pattern box. I started off using deployed-build-%system.build.number% as my tag which just marks git with the TeamCity build number.

When I run a TeamCity deployment I don’t always use the same configuration options, I deploy locally, to a test server or to production and sometimes I just deploy the frontend or the backend. How cool would it be to include this information in the tag text?

Well the next step was to change my Labeling pattern to deployed-build-%system.build.number%-%ServerName%-%DatabaseName%-%FrontEndPath%, this adds the backend database config settings and the path the frontend was deployed to. Now when looking at git you can see commits marked with multiple tags, one for each deployment that succeeded and the tag will indicate the settings used during that deployment.

Now I will never forget to add the tag after a release as the adding of a tag is part of the deployment process, if the deployment fails the tag won’t be added. I can test my deployment to test and git will show if this has been successful, and when I deploy live this will also show up.

How do you use tags? Do you mark successful builds with a tag? Why not let me know or leave a comment below.

I’m 100 blog posts old

funkysi1701@gmail.com (funkysi1701) — Thu, 02 Jun 2016 20:00:45 +0000

That’s right this is the one hundredth post that I have written on this blog.

So what have I learned in the past 100 posts?

It is easier to write a blog before you become a parent. More recently I am increasingly finding it difficult to find the time to blog. It used to be that I could write on an evening, but now James is around I often prefer to play with him, or more often stop him crawling where he shouldn’t.
I like my job. The inspiration for most of this site is my day job and as you can read, I do a wide variety of different things, but I have a lot more to learn as well.
Finding your niche is hard. 100 posts in and I am still not sure what my niche is. I started out with the broad niche of IT and what I do, I then considered something about IT and fatherhood but I don’t think that topic is really me. My current thinking is maybe DevOps especially as my role these days fits squarely between Development and Operations.
Its time for a refresh. I have been meaning to change the theme of this site for some time and I feel after 100 posts now is as good as time as any. I want to emphasise my skills and what I am learning and increase the emphasis on DevOps, which I think will be my niche.
Watching visitor numbers is addictive. Every day I look at how many people have looked at my blog, but I have yet to see a pattern between what I write and how many reads I get. Is my writing getting better? I don’t know. Are more people reading? Probably not. Will I keep going? Yes

So what is next?

Hopefully a refreshed look in the next few months. Hopefully regular posts. If there is something you want to see on here drop me a message via any of the social media links or put a comment below.

What is my favourite post?

Maybe User Groups and F# which proved very popular and got me to start going to user groups something I still enjoy today. I also like Coding Myself Into A Corner which got me to start thinking more if I was giving myself future problems. But there are many others I like such as James and Becoming a father

TNG Brothers S4 E3: Battle Bridge

funkysi1701@gmail.com (funkysi1701) — Thu, 26 May 2016 20:00:45 +0000

My latest podcast can be found at https://www.trekmate.org.uk/brothers-tng-s4-e3-review-the-battle-bridge/ .

Data jeopardizes an emergency mission to save an ill child when he gets a signal from his creator.

Today James and Lou discuss (Brothers) and what we think of it, with tonight’s guest host Simon Foster, Funky Si.

Credit: Main Title Theme (TV Edit) was arranged by Dennis McCarthy and composed by Jerry Goldsmith and Alexander Courage

Running Windows on Raspberry Pi

funkysi1701@gmail.com (funkysi1701) — Thu, 05 May 2016 20:00:45 +0000

Last year you may remember me talking about playing with a Raspberry Pi. Well since then my Raspberry Pi has been sat on a desk collecting dust.

This week I attended Leeds Sharp and the topic was Running Windows on Raspberry Pi and this has inspired me again to do something with a Pi.

But first what did I learn.

Here are a couple more of pics of last nights @LeedsSharp https://t.co/QvlsYjNFvB #RaspberryPi #MSIoT pic.twitter.com/60o4wiiPSv
— Richard Tasker 🇬🇧 (@ritasker) April 29, 2016

I had heard that a cut down version of Windows 10 could be installed on the newer Raspberry Pi’s, but I hadn’t really understood how cut down the version of windows is. Having now seen it demonstrated the OS consists of a single page with a few menu options.

The real power of Windows 10 IoT is when you connect remotely to it. There are a couple of ways to do this, PowerShell (check out https://github.com/MicrosoftDocs/windows-iotcore-docs/blob/main/windows-iotcore/manage-your-device/CommandLineUtils.md for a few commands), and of course connecting Visual Studio to your Pi.

When I had previously played with a Pi, it had been with bash scripts and linux commands. The beauty of installing Windows IoT is that you can write c# code, something I do in my day job so theoretically I should find it easier.

The demonstration at Leeds Sharp was pretty impressive. If you are a fan of the Big Bang Theory you may recall Sheldon playing a Theremin. Well it is actually possible to construct a Theremin from a couple of sensors and a Raspberry Pi. The code for which is on github .

Now that I have been inspired what shall I do?

My Raspberry Pi won’t support Windows 10 IoT, so I need to buy the latest version. I am thinking of buying a kit so I can play about with a breadboard, LEDs and resistors. Maybe not build a robot straight away but certainly try doing something that connects to the GPIO pins.

If you have any suggestions leave a comment below.

Exchange Web Services

funkysi1701@gmail.com (funkysi1701) — Thu, 21 Apr 2016 20:00:45 +0000

Where I work we use a really old fashioned way of recording where in the country employees are: Excel!

For years I have been trying to persuade staff to use calendars in Exchange. Outlook is great for looking at one or two people’s calendars at once but quickly gets unmanageable for looking at ten or more peoples availability.

Recently I have started looking into how easy it is to query this information to give a custom view.

Microsoft provide an API to query exchange information called Exchange Web Services or EWS. I have only used EWS with my Exchange 2010 setup, but the documentation mentions working with Exchange 2007 and older or Exchange online.

Here are the basics of what I have tried. Fire up Visual Studio and from nuget install EWS.

Install-Package Microsoft.Exchange.WebServices

I started off with a simple Console App to see how it all worked, and then extended it to a MVC website. I found querying exchange directly was slow, but it is easy enough to cache information in a database.

To start you need to create an Exchange Service object, by specifying the version of Exchange you are using.

ExchangeService service = new ExchangeService(ExchangeVersion.Exchange2010);

Next you need to pass the URL you are using to access Exchange.

service.Url = new Uri("mail.example.com");

To access information from exchange you need to pass some Exchange credentials, ideally a username and password that has access to view all the calendars you want to look at.

service.Credentials = new WebCredentials("username","password");

Next pass the email address of the user who owns the calendar you want to look at.

service.ImpersonatedUserId = new ImpersonatedUserId(ConnectingIdType.SmtpAddress, "me@example.com");

My particular exchange server has a self signed SSL certificate which is not going to be trusted by remote clients. The following line ignores this validation check and makes my program connect.

System.Net.ServicePointManager.ServerCertificateValidationCallback = (sender, certificate, chain, sslPolicyErrors) => true;

Now that we have connected to exchange we just need a few lines to look for events in the calendars.

// Initialize the calendar folder object with only the folder ID.
CalendarFolder calendar = CalendarFolder.Bind(service, WellKnownFolderName.Calendar, new PropertySet());

// Set the start and end time and number of appointments to retrieve.
CalendarView cView = new CalendarView(startDate, endDate);

// Limit the properties returned to the appointment's subject, start time, and end time.
cView.PropertySet = new PropertySet(AppointmentSchema.Subject, AppointmentSchema.Start, AppointmentSchema.End);

// Retrieve a collection of appointments by using the calendar view.
FindItemsResults<Appointment> appointments = calendar.FindAppointments(cView);

Now that you have an appointments object you can loop through each element with a foreach loop. In my case I assign each elements Subject to a variable, which I can then do what I like with (display in a console window, save to a database, display in an MVC website.)

foreach (Appointment a in appointments)
{
 if (a.Subject != null)
 {
 subject += a.Subject.ToString();
 }
}

My website queries a SQL database which I can easily populate with a console app that runs at regular intervals throughout the day.

There is a lot more I want to do with this project as this is only the basics of what you can do with Exchange Web Services. So expect more blog posts on this subject as I expand its functionality and learn new ways of doing things.

Revisiting Team City

funkysi1701@gmail.com (funkysi1701) — Thu, 24 Mar 2016 20:00:45 +0000

Last year I blogged about Team City , well I have been looking at it again recently. In that time they have even changed their logo!

Lets start with thinking about what I want my Continuous Integration server to do.

Check out my code from source control (usually master but all feature branches would be even better)
Configure specific setting for build
Build my code
Build my databases
Run any unit tests
(Optional) Run deployment to Azure Test/Live site

There are probably other things I want to achieve but I will start with these six.

Checking out code from source control is something Team City does out of the box, so I can safely say I have done this now. It even monitors a branch for changes and initiates a new check out.
Team City allows you to create specific build steps so in theory you can have multiple builds for every variation of settings that you want for your code. I have not tried this yet apart from building with the default config, but I don’t expect it will be too difficult.
I have managed to get my code to build with Team City, it took a bit of tweaking the different build steps but wasn’t too difficult. Team City has a visual studio build agent which takes you solution file and does what it needs to. The one problem I have found with this step is that I get errors with my tests if I select a Debug config instead of Release.
Databases are always the problem part of the deployment. So far I have manually deployed my databases but I intend on revisit this step. A stackoverflow post suggests that I can run SQL code via Team City in the following way by creating a command line executable:

Command executable: c:\Program Files\Microsoft SQL Server\100\Tools\Binn\sqlcmd.exe
Command parameters: -S [ServerName] -i [PathToSQLScript]

I have yet to try this but I am hopefully that it will just work. Dropping a database and restoring a back and then running different SQL scripts is all possible from TSQL, so I should be OK. Watch this space for more details.

Running the unit tests got me stuck for a while. I tried setting it up using VSTest or MSTest neither worked mainly because a config file wasn’t being copied with the test binaries. When I tried using NUnit it just worked. The tests that failed gave me a few config settings to change.
I have powershell scripts that deploy to Azure websites, I think that these could form the basis of a deployment to Azure. Again the difficult step here may end up being deploying all the different databases to Azure. This is also the riskiest step as I need to connect to live servers which is why I will leave this to last, at the very least I could generate scripts that do a full deployment.

That’s it for now. Once I have this all working I will revisit again with details of the database steps as I am expecting a few challenges to overcome. What have you used a CI Server for? Are there other things I want to achieve from a project like this? Why not contact me or leave a comment below

Model View Controller (MVC)

funkysi1701@gmail.com (funkysi1701) — Thu, 17 Mar 2016 20:00:45 +0000

Model View Controller or MVC is a software architectural pattern for implementing user interfaces on computers. It divides a given software application into three interconnected parts, so as to separate internal representations of information from the ways that information is presented to or accepted from the user.

I have been trying to get my head around the concept of MVC for a while, hopefully writing this article will help solidify my understanding of it.

One of the core concepts of MVC is the ability to separate concerns so you can concentrate your energies on one aspect of the application.

Model This is the data. If your application uses a database the model often mirrors what you have in the database and concerns itself retrieving information from the database.

View This concerns itself with displaying the data to the user. Typically this is the html pages of your application.

Controller This concerns itself with actually doing things and deals with user interaction. Typically it will get data from the view and send data to the model.

The three concerns can be developed in isolation as they do not depend on each other, for larger development teams you can even divide up development much easier that a traditional app.

The basic concept of MVC I get and understand, however I find myself getting bogged down in the details.

The database doesn’t matter. I need to remember this and not get sidetracked in writing custom methods to connect to the database which end up unmanageable. I know SQL, so can easily write SQL commands to copy data into the format I need for my app. The app I am currently working on involves a large amount of existing data, and I need to concentrate on the MVC parts and worry about the database later.

In previous attempts I have tried to build my model against many tables, but instead I can write a query against many tables and insert that into one table which the Model can then use.

Changing the model often results in an error informing you that the context has changed since the database was created. The easy solution to this in my case is to drop the database and allow entity framework to recreate the database each time. As long as my database contains no new data, I won’t loose anything.

One of the core advantages of MVC is the ability to test it or even use test driven development (TDD). I haven’t really dabbled with testing yet as I am still trying to get my head around the fundamentals, but once I have made some progress with my app I want to test, so next time I get asked to add a new feature I have no fear about breaking stuff.

For the first time I have got an app with a working Authentication system from the start. And it is remarkably easy to implement with one keyword. Adding [Authorize] to the top of your controller is all that is needed. Building the actual Authentication system is relatively easy from Visual Studio, as it has templates for Azure AD, Forms Based, Open Auth like google/twitter etc.

It is still very early days for my MVC app and my understanding of it, but I feel I have turned the corner and can actually build something with it now, rather than be stuck in a downward spiral of confusion.

What do you think about MVC why not leave a comment below? For more info about MVC I have been looking at https://dotnet.microsoft.com/en-us/apps/aspnet which has more information and tutorials.

Surface Book Review

funkysi1701@gmail.com (funkysi1701) — Thu, 10 Mar 2016 20:00:45 +0000

I have just bought myself a new laptop, but it is not just any laptop it is a Microsoft Surface Book .

I think this is probably the first time I have bought myself a top of the range laptop and after a few days of use I am loving it.

The Surface Book is the latest in Microsoft’s Surface line of tablets but the first to feature a keyboard and be more like a traditional laptop. The keyboard is detachable from the keyboard so you can use the Surface Book like a tablet.

As it is a top of the range laptop, it was not cheap, so if price is a significant factor in your laptop choice this is not the machine for you. Microsoft are trying to compete with Apple’s Macbook range.

So what did I get for my money:

Windows 10 Pro
16Gb RAM
512Gb SSD (Formatted Size nearer 474Gb)
6th Generation Intel Core i7 Processor
Surface Pen
13.5-inch touch screen

I have only had the device for a few days but these are my thoughts so far. My primary reason for buying this is for doing development work with Visual Studio and SQL Server.

I really like the Surface Pen. This is a feature that I didn’t think I would use much. The Pen connects via bluetooth and allows interaction with the touch screen. The pressure sensitive screen allows all sort of touch actions to be performed.

I especially like the choice of keyboards that are offered when the physical keyboard is detached. You can have either a on screen keyboard which you can type with pen or fingers, an on screen keyboard that splits in half or you can write with the pen. This is an amazing feature. My handwriting is not good but most of the time it understands my scrawl. With this option you can hand write a tweet or fill in a form on a webpage. For speed I would not recommend this form of input for large amounts of text, but for browsing the web of when you are just clicking on links it is great.

Windows Hello – I can unlock windows just by looking at my screen. How cool is that! It was really easy to setup, it just takes a photo of your face and next time you login all you need to do it look at the screen. Note if you are working in tablet mode, make sure the tablet is the correct way up.

It’s not all brilliant though. Detaching the screen is fiddly and sometimes takes a few moments to do. A few times I have felt I needed another hand but I am sure the more I do this the easier I will find it to do.

Battery life isn’t great especially when running off the tablet only. This is due to the machine having two batteries, one in the base and one in the screen so with keyboard attached you have much longer use times. Actually I am finding the battery is lasting longer now that I have used it for a few days.

The screen has a very high resolution 3000 x 2000 but using clever zooming technology everything is still readable and not tiny. However I use Remote Desktop a lot and this caused me a problem. When RDPing the remote session used the host screen resolution which made everything tiny on my servers. The solution to this can be found on SuperUser and involved using Microsoft’s Remote Desktop Connection Manager, a cool bit of software for managing multiple RDP sessions. This is actually an improvement on the way I usually work, but until I found the answer this was annoying.

Another minor annoyance with the keyboard is that you can’t press Ctrl-Alt-Del with one hand. Normally this isn’t a requirement but if the screen locks while I have my son on my lap I am stuck, but I won’t blame the surface book for this.

When I got my Surface Book I was also given a Microsoft Wireless Display Adapter. I am not a fan of this bit of tech as I can’t get it to work. I briefly got my old laptop to connect but my Surface Book keeps telling me NO!

Overall I like the Surface Book. It is certainly the nicest laptop I have ever used. Now that I have the docking station I can connect two monitors, have a wired internet connection and it becomes a proper work horse. What does everyone else think? Better than Apple’s range of laptops?

Trying Out Azure Active Directory

funkysi1701@gmail.com (funkysi1701) — Thu, 03 Mar 2016 20:00:45 +0000

One of my plans is to create new MVC Webapps for my companies databases. Once I publish these I will need to secure them so only staff have access.

The traditional way to do this would be insert membership tables into my database. The user then has to remember another username and password and I have to secure the storage of these credentials. Lots of work for everyone.

There is a better way by using Azure Active Directory. You have probably heard of Active Directory, if you are a SysAdmin you probably use it all the time to manage your corporate users and computers. Azure Active Directory is an extension of this into the Cloud.

I have blogged in the past about using Azure but this is the first time I have tried connecting my internal domain to Azure. There is a Virtual Lab which helped me try out some of these ideas.

The first thing I did was to create a new Directory in Azure. I did this via the old portal (manage.windowsazure.com ) it might be possible via the new portal but I don’t know how yet.

Click New, select App Services > Active Directory > Directory and select Custom Create. Select Create new directory, give it a name and a domain name and select a region from the drop down. Then add a Global Admin for this directory.

There is a tool called Azure Active Directory Connect . Download and Install this with express settings on one of your domain controllers. You need to specify a domain admin account to access your domain and the Azure Global Admin account you just created.

At this point I went to bed so I am not sure how long it tool to sync all the domain information but by morning it was all showing in the users list on Azure.

All my user accounts are showing with a @contoso.onmicrosoft.com, it is possible to use custom domains but I haven’t figured out that step yet. I made a change in my Active Directory and a while later that change was showing in Azure AD.

So now what? Open up Visual Studio and see if I can use Azure to Authenticate.

I selected to create a new MVC web project and clicked the change authenticate option. One of the options was Work and School Accounts, I then selected Cloud Single Organization and entered contoso.onmicrosoft.com. I then ran this app and it authenticated using Azure using my domain password. Really impressed at how easy that was.

The app then shows up on Azure in the old portal. In Applications you can see a list of which users have access to your app and configure few other app related settings.

This is a long way off being useful in my actual app, but it shows that the basics of what I am trying to do does work. Anyone done anything similar with Azure AD? How did you get on?

Coding myself into a corner

funkysi1701@gmail.com (funkysi1701) — Thu, 25 Feb 2016 20:00:45 +0000

I spend an awful lot of my working life adding new features and improvements to a legacy database system.

A legacy system is an old method, technology, computer system, or application program, “of, relating to, or being a previous or outdated computer system.” Often a pejorative term, referencing a system as “legacy” often implies that the system is out of date or in need of replacement.

My particular legacy system is Microsoft Access. We use Microsoft Access to provide the front end for all our internal databases. The particular technology we use is ADP files which are only supported in Access 2000, 2003 or 2010.

Microsoft has already dropped support for Access 2000 and 2003, and support for 2010 is due to be dropped in 2020.

So why am I coding myself into a corner? Its simple for every feature or improvement I create in Access, I increase the amount of features I need to create or port to a new front end.

This is a very depressing thought, everything I create will need creating again. I am not saving myself work but increasing the amount of work I need to do again.

The bad news is that my employers like almost all businesses demand results and like a good employee I have been delivering them. I have been promised the mythical “when we are quiet” you can work on rebuilding the database front end. I know this will likely never happen so what are my options.

Do nothing I have warned my managers that this work needs doing and it is up to them to give me the resources I need. This is not an acceptable option. Firstly I am not future proofing the business, 2020 will be here before we know it. Also I am not developing myself as a developer, as the only experience I am getting is with Legacy technology that has expired or will do very soon. No one wants to employ someone who only has legacy experience.

Do something I need to keep delivering results and doing what is asked of me. Like any good engineer I should be multiplying my estimates by four, meaning that I have some time that can be used for looking at the bigger picture. This is a win-win option really. Employer gets a solution that is future proof, Employee gets valuable experience in up to date technology.

What do you think? Have you been tied to legacy technology? Why not leave a comment below.

Trek review: Best of Both Worlds

funkysi1701@gmail.com (funkysi1701) — Thu, 18 Feb 2016 20:00:45 +0000

Episode Title: The Best of Both Worlds
Star Trek Type: TNG
Original Air Date: 18 June 1990 & 24 September 1990
Written By: Michael Piller
Directed By: Cliff Bole

The Borg threaten the Federation, when they kidnap Captain Picard and destroy a fleet of 39 Starships. Luckily some original thinking from Commander Riker and Lieutenant Commander Shelby saves the Federation from assimilation.

Thoughts: I love this episode! It is probably my favourite episode in the whole of trek. I can’t think of a single scene which I don’t like.

It is a two parter that actually works best as a two parter. The moment that Riker orders the Enterprise to fire on the Borg is great. At the time some thought that Patrick Stewart might leave the show and be replaced by Riker/Shelby. I am very glad that didn’t happen but it is interesting to ponder what that might be like.

The Borg soundtrack in this episode is great. Ron Jones really captures the spooky threat that the Borg are. I was always disappointed that some of that didn’t continue in First Contact and when the Borg return in Voyager.

The episode has a cracking pace. From the moment the Borg first attack the Enterprise to the moment the Borg ship explodes the show doesn’t slow down.

I am Locutus… of Borg. Resistance… is futile.
Your life, as it has been… is over. From this time forward… you will service… us.

Commander Shelby is a great character that enhances the episode and introduces conflict between characters, particularly Riker.

I have heard that Riker’s thoughts about his career was a mirror to writer Michael Piller’s thoughts about leaving the show. Luckily for us he stayed.

In a future episode of DS9 Worf comments that he felt there was nothing that they couldn’t do during this time. I have to agree the tag team of him and Data is a great one. If I want rescuing it would be Data and Worf that I would want.

My good friends at Trekmate have just reviewed part one, why not have a listen?

The Best of Both Worlds along with all 900+ Star Trek episodes is now available in Episode Atlas .

My git repository is too large!

funkysi1701@gmail.com (funkysi1701) — Thu, 11 Feb 2016 20:00:45 +0000

Today I did a clone of one of my git repositories and it took ages to download. Looking into what got downloaded it was easy to see why. The .git folder was over 500Mb in size.

I know how this has happened. This repository was created in 2013 and has been used by me as a dumping ground for lots of things related to the code which never should have been committed.

Since 2013 I have learnt a lot more about coding and git so the current version of the files in git isn’t too bad. But git keeps the history of changes for every file so bad practices like this are kept.

What can I do about this? Well what does google suggest? I found this https://stevelorek.com/how-to-shrink-a-git-repository.html (link no longer exists).

It suggests ways of listing all the large files that are stored in git and a way to remove them. As I am the only person that regularly commits to this repository I see no problem with giving it a go.

I will summarise the steps here.

git clone url

Now you need all the remote branches so there is a bash script to run

#!/bin/bash
for branch in `git branch -a | grep remotes | grep -v HEAD | grep -v master`; do
git branch –track ${branch##*/} $branch
done

Another bash script then lists the top 10 large files

#!/bin/bash
#set -x
# Shows you the largest objects in your repo’s pack file.
# Written for osx.
#
# @see https://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/
# @author Antony Stubbs
# set the internal field spereator to line break, so that we can iterate easily over the verify-pack output
IFS=$’\n’;
# list all objects including their size, sort by size, take top 10
objects=`git verify-pack -v .git/objects/pack/pack-*.idx | grep -v chain | sort -k3nr | head`
echo “All sizes are in kB. The pack column is the size of the object, compressed, inside the pack file.”
output=“size,pack,SHA,location”
for y in $objects do
# extract the size in bytes
size=$((`echo $y | cut -f 5 -d ‘ ‘`/1024))
# extract the compressed size in bytes
compressedSize=$((`echo $y | cut -f 6 -d ‘ ‘`/1024))
# extract the SHA
sha=`echo $y | cut -f 1 -d ‘ ‘`
# find the objects location in the repository tree
other=`git rev-list –all –objects | grep $sha`
#lineBreak=`echo -e “\n”`
output=“${output}\n${size},${compressedSize},${other}”
done
echo -e $output

After running the script you will see details of your largest files. I had *.msi and *.exe files in the mix. To remove them run the following, where filename is the path to the file that needs removing.

git filter-branch –tag-name-filter cat –index-filter ‘git rm -r –cached –ignore-unmatch filename’ –prune-empty -f — –all

To reclaim the disk space run the following commands

rm -rf .git/refs/original/
git reflog expire –expire=now –all
git gc –prune=now
git gc –aggressive –prune=now

Now push your changes back to the remote server.

git push origin –force –all
git push origin –force –tags

Now if you do a clone it will be much smaller than before and you can get back to coding much quicker, without having to wait.

SQL Transaction Log Backups

funkysi1701@gmail.com (funkysi1701) — Thu, 04 Feb 2016 20:00:45 +0000

Like many DBAs I spend a lot of time maintaining my SQL Server backups.

From SQL Server I maintain both full backups and transaction log backups. I have often restored my full backups but until recently I have never restored a transaction log backup. All backup strategy’s are only as good as the last time you tested the restore process.

So what is a transaction log backup?

A transaction log backup contains all the transaction log records generated since the last full backup and is used to allow the database to be recovered to a specific point in time (usually the time right before a disaster strikes). Since these are incremental, if you want to restore the database to a particular point in time, you need to have all the transaction log records necessary to replay database changes up to that point in time.

How to do the restore.

First right click on Databases in SQL Management Studio and select restore database. You should then get a screen similar to this.

In source click the … to allow you to select your backup files.

Now normally I have only ever selected one file here, the *.bak file. Instead select the *.bak and all the *.trn files as well. After SQL Server has chugged for a few minutes (time will depend on number of transaction files and server/disk speed etc) the restore plan section should fill up with files.

In the destination database box, type in the name of the database you want to restore. I recommend using a different name to avoid overwriting the original database, appending Test or a datetime to the name is what I usually do.

On my test server I need to untick the take tail-log backups option off the options screen before I can execute the restore.

Now you can either check the tick boxes in the restore plan section or (more fun) click the timeline button to select at what point in time you want to restore to.

You can either select the point in time with your mouse or specify the exact point in the time textbox. Alternatively you can just select the most recent point, probably the most likely option when disaster strikes.

Now that I have tried doing this on my test server I feel much more confident that when disaster does strike I can get things restored quickly and painlessly.

How often should you run transaction backups?

The answer to this question depends on how critical your data is. Until very recently I ran mine ever 15 minutes, I have increased this to every 5 minutes, but I have seen recommendations of running it every minute . The more critical your data the more often you should run them.

Development Annoyance

funkysi1701@gmail.com (funkysi1701) — Fri, 29 Jan 2016 20:00:45 +0000

I spend a lot of my time creating new features that simplify my companies business processes.

A good example of this is an invoicing system I created. Instead of users working off different spreadsheets and copying and pasting data between various different programs the user can click a couple of buttons and everything is done and they can move on to the next task.

I am currently expanding this system so more of the companies work can be invoiced quickly and simply.

I like working on this kind of problem

I get to discuss with the different departments involved, finding out how they work and what could make their lives better.

I then get to look at what structures already exist in the database and get to rip out anything that makes no sense (Today I had a Customer table which has a primary key called ClientId – its now a Client table with PK ClientId) and add new structures to store new information.

I then can build a user interface so the users can interact with the data.

But the most important part of the process comes next. This is when I show the users what I have built and how it will make their lives better, they can then feed back to me questions and comments. How do I do x?, what about y?, have you thought about z?

After this very valuable feedback I can go back and tweak what I have done making it better and better until the users think it is suitable for what they need.

Now why did I call this post development annoyance?

I find it intensely annoying that after spending weeks of my time working on creating something to be used. Users either make do with my solution and never tell me the one thing that really annoys them or worse they go back to the time consuming process and never feedback the one or two changes I need to make to complete the project.

Now I am not perfect, like many IT people my people skills are somewhat lacking but apart from asking people what can you do to get the feedback you need to complete the project?

I do not have the knowledge that other departments have so I consider development a team effort that requires everyone to contribute otherwise it will not finish.

The second outcome where users return to their old ways is the most annoying. It could well be months or even years before I find out that what I created is not being used any more. By that time my memory of what I did and more importantly why I did what I did has been lost and forgotten.

So users please, please tell me what works and what doesn’t and what things I can do to improve things. I want my solutions to constantly improve and get better and I can only do that with your help.

Star Trek Episode Review Arena

funkysi1701@gmail.com (funkysi1701) — Thu, 21 Jan 2016 20:00:45 +0000

Episode Title: Arena
Star Trek Type: TOS
Original Air Date: 19 January 1967
Teleplay: Gene L Coon
Story By: Frederic Brown
Directed By: Joseph Pevney

The episode begins with the Enterprise in orbit of Cestus III. Commodore Travers requests a tactical team join the away team. When the away team beam down Cestus III has been destroyed.

Spock detects non-human life signs, obviously the aliens that destroyed the outpost. They attack the Enterprise, stranding the away team who are under heavy fire. Sulu is instructed to do whatever he can to protect the Enterprise and leaves orbit.

Kirk finds the armoury and a grenade launcher, after firing the enemy withdraws.

After beaming back aboard the Enterprise they pursue the unknown enemy into uncharted space. One of the survivors of Cestus III explains that the enemy attacked without warning.

Kirk is determined to destroy the enemy ship before it reaches home and the Enterprise prepares for battle.

An unknown solar system scans the Enterprise as they continue their pursuit of the enemy ship. The enemy ship comes to a complete stop. The Enterprise prepares to attack but before they can they also come to a complete stop. The Metrons have stopped both ships because both ships were on a mission of violence. The captains of both ships will be placed on a planet where the winner can destroy the other.

Captain Kirk and the enemy captain appear on a desert planet. The enemy is a reptilian like species called the Gorn. The Gorn has superior strength to humans but Kirk manages to escape his first encounter.

"This is Captain James Kirk of the Starship Enterprise.
Whoever finds this, please get it to Starfleet Command.
I’m engaged in personal combat with a creature apparently
called a Gorn. He’s immensely strong. Already,
he has withstood attacks from me that would have
killed a Human being. Fortunately, though strong,
he is not agile. The agility and I hope the cleverness, is mine."

The Metrons said that the planet contained materials to allow the building of a weapon, however at first glance Kirk sees nothing that could help him defeat the much strong Gorn captain. He does find some diamonds, too small to be used as a weapon.

Kirk pushes a huge boulder over a cliff onto the Gorn captain, which appears to knock him out, however he survives and Kirk barely escapes. Kirk then finds Sulphur, which triggers a memory.

The Metrons allow the Enterprise to watch the last stages of Kirks struggle. They see Kirk discover Potassium Nitrate. The Gorn accuses the Federation of trespassing into their space.

Kirk starts constructing a weapon using the diamonds, sulphur and potassium nitrate to make a gunpowder explosion to fire the diamonds at the Gorn captain. Just in time he applies a spark to his weapon and his opponent lies defeated.

Kirk refused to kill the Gorn captain as they may have been just defending themselves. The Metron announces that Kirk has displayed the advanced trait of mercy and there may be hope for humanity after all.

Thoughts: I like this episode very much. It has some classic Star Trek ideas in it.

We have a super powerful alien race (the Metrons) that think there is hope for humanity.

We have Kirk using raw materials to construct a weapon using simple chemistry.

We have an enemy race that appears destructive but may have just been defending themselves.

We have the classic location of Vasquez Rocks, somewhere that if I ever visit the US I have to visit.

We have the very cheesy Gorn mask, yes its not realistic but the story is strong enough that it doesn’t really matter. Enterprise brought back the Gorn but as a CGI character and I think I may prefer the rubber mask.

Interestingly the weapon Kirk builds would likely not work. The US show MythBusters proved that the bamboo wasn’t strong enough to hold the explosion and the weapon would likely wound Kirk as much as it would wound the Gorn.

Score: 10/10 Redshirt Count: Cestus III Outpost, 1 Red Shirt and 1 yellow shirt from away team.

Arena along with all 900+ Star Trek episodes is now available in Episode Atlas .

Disaster Planning

funkysi1701@gmail.com (funkysi1701) — Thu, 14 Jan 2016 20:00:45 +0000

Over the Christmas break the city of York experienced the worst flooding disaster it has seen in recent years. If you want to help donations are being taken at York Flood Appeal .

Luckily I wasn’t affected but it was really sad seeing many of my favourite streets full of water. Thousands of people had to be evacuated from their homes when the water was at its highest.

York has its phone exchange located in the centre of the city and as the flood waters rose, this building flooded. BT who run the exchange worked flat out trying to restore phone and broadband services to York and the surrounding area, but this lead to at least 24 hours or more of downtime for users.

During this time most of the city wasn’t able to take card payments from customers (due to an internet connection being required), cash machines were not working for the same reasons and the most worrying thing access to emergency services was not available.

Anyone would think the city phones were down #yorkfloods #bthardatwork pic.twitter.com/ZavLMsG2Js
— Simon Foster (@funkysi1701) December 28, 2015

I am in no way critical of the hard working engineers that worked on the BT exchange to restore services to the city. However I have to ask what was contained in the BT disaster recovery plan.

As an IT Manager myself backups and disaster planning is something I need to think about and plan for. Recent investment in our internet connection meant that we are now on a leased line so BTs downtime would not have affected us in terms of network connectivity, I am uncertain about phone but I believe as a business we could have continued running during this time if we needed to.

Planning for a disaster is not something I ever make enough time, however I definitely want to spend more time on it and I do have the basics of a plan in my head which is probably more than BT has.

BT is a huge company they run almost the entire country’s phone lines and according to wikipedia have operations in 170 countries. I think that they can afford to have someone plan for disaster recovery.

The York Exchange is located next to a river that often floods. What could the company have done to avoid this problem?

Relocate some or all of its services to a location away from the river.
Add the ability to reroute services through an alternative exchange.
Many towns and villages surrounding York use the BT exchange surely some of these could have been configured to use other exchanges.

In these days of terror alerts and bomb scares. Imagine if York was London and all it took to take down communication ability of the capital was one Exchange to be offline?

I am sure BT are looking at what they did right and what they did wrong and if you are reading this have a look at your own disaster recovery policy. Don’t get caught out like BT did.

Top 50 Star Trek episodes

funkysi1701@gmail.com (funkysi1701) — Thu, 07 Jan 2016 20:00:45 +0000

In 1966 a TV show called Star Trek first aired in the USA. Since then 726 episodes have been made and 12 films have been released. 2016 marks the 50th anniversary and Star Trek is as alive as ever with a new film due out this summer and a brand new TV series in the works for broadcast in 2017.

But what can I do to celebrate? With that many episodes I don’t have enough time to re-watch every episode so how about a re-watch of the top 50 episodes.

Star Trek has five live action TV series so all I need to do is pick my top 10 from each.

Star Trek

The original Star Trek series first aired on September 8th 1966. It lasted only three years and 79 episodes. This series featured the iconic characters Kirk, Spock and Bones on a five-year mission to explore space.

The City on the Edge of Forever
The Devil in the Dark
Balance of Terror
The Trouble with Tribbles
Arena
This Side of Paradise
Amok Time
Space Seed
Mirror Mirror
The Deadly Years

Star Trek: The Next Generation

The first spin-off series set 100 years after Kirk featured a new Enterprise continuing to explore space. This series aired between 1987 and 1994 for 178 episodes.

Best of Both Worlds
Yesterdays Enterprise
The Offspring
The Measure of a Man
Disaster
All Good Things…
Tapestry
The Inner Light
Cause and Effect
Unification

Star Trek: Deep Space Nine

The second spin-off series which ran at the same time as TNG was set on a space station so instead of exploring adventures arrived via a stable wormhole. This series aired between 1993 and 1999 for 176 episodes.

Emissary
Duet
The Way of the Warrior
The Visitor
Sacrifice of Angels
Waltz
Trials and Tribble-ations
In the Pale Moonlight
Far Beyond the Stars
Siege of AR 558

Star Trek: Voyager

The third spin-off series ran after TNG and featured a female captain and a starship stranded in a remote part of the galaxy journeying home. This series aired between 1995 and 2001 for 172 episodes.

Scorpions
Year of Hell
Caretaker
Message in a Bottle
Eye of the Needle
Heroes and Demons
Jetrel
Timeless
Pathfinder
Riddles

Star Trek: Enterprise

The most recent spin-off ran after Voyager and was set 100 years before Kirk. This series aired between 2001 and 2005 for 98 episodes.

Broken Bow
The Andorian Incident
Shuttlepod One
In a Mirror Darkly
Twilight
Damage
Azati Prime
Regeneration
Impulse
Zero Hour

I am not going to review each episode here, I will save that for future blogs. With 50 episodes that is one a week to get them all reviewed during Star Trek’s birthday year. We will see if I manage to keep to that schedule.

Track all these episodes and more in Episode Atlas .