2015-04 on Funky Si's Blog (Dev)

I might actually like SQL Server

funkysi1701@gmail.com (funkysi1701) — Thu, 23 Apr 2015 20:00:45 +0000

How did this happen I think I may actually like SQL Server now?

I remember a few years back when I used to grumble about writing a SQL query that included a JOIN. For some reason back then I could never get my head around JOINs.

Today is a totally different story. I wrote a deployment script that did the following:

dropped a database,
restored a database from a backup file,
Created a database table,
copied the contents of a table into this table,
disabled and re-enabled some triggers,
dropped some constraints and columns from a table
and renamed some columns.

It’s not like I woke up this morning with SQL Server knowledge. For many years I have been adding extra features and changing functionality to our line of business database and over time my SQL confidence has grown.

For every single one of the steps above I googled and looked up the SQL syntax (every time I write an insert or update statement I look up the syntax, one day it will stick in my brain!) I think the main reason is once you have used SQL for a while you get to see how it works and can split it down into small steps.

As I blogged the other day I am currently working on improving a bad database and today I wanted to test the deployment process. All my changes are in a SSDT project so I took a backup of my database and tried to publish.

Error! Your changes will result in data loss, no surprises there as I was expecting that error. The main culprit for this was a trigger I wrote but I didn’t find that out until the end of the day.

As I was working on a backup I could do what I like including destructive changes. I tried creating a pre deployment script. I thought if that runs first, I can persuade SSDT that it matches SQL Server and therefore no data is being lost.

This didn’t work, but I had created the start of my script mentioned above. I had steps 3 and 4. Lets try running my script first and then try running the deployment of my SSDT project. To get this to work I am going to have to rebuild my database a few times, I did this manually with SQL Management studio a couple of times until I thought, I could add this to my script – that’s step 1 and 2 done.

Then I got some errors about triggers so I tried disabling them and re-enabling them after it had finished. That’s step 5.

Finally I needed to drop some columns, this resulted in an error about a constraint. A few trial and error run through’s to find out which constraints I needed to drop and then I could drop the columns.

The last step was to rename some columns. I thought after this I can run publish from visual studio. No it still complained about data loss. I dropped another trigger and then I could run everything without any errors.

Wohoo! Aren’t I clever. A simplified version of my code is below

USE master -–can’t drop the database if its open!
–-Step 1
DROP DATABASE DBName

–-Step 2
RESTORE DATABASE DBName FROM DISK=’E:\DBName.bak’
WITH
MOVE ‘DBName_dat’ TO ‘C:\Program Files (x86)\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\DBName.mdf’,
MOVE ‘DBName_log’ TO ‘C:\Program Files (x86)\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\DBName.ldf’,
REPLACE,
STATS=10
USE DBName –-swap back to the database you just restored

–-Step 3
CREATE TABLE TName2 (
ID INT NOT NULL,
Name NVARCHAR(50) NOT NULL,
Address NVARCHAR(50) NOT NULL,
City NVARCHAR(50) NOT NULL,
Postcode NVARCHAR(50) NOT NULL)

-–Step 5
DISABLE TRIGGER TR_Trigger ON TName1
GO

–-Step 4
INSERT INTO TName2 (Id,Name)
SELECT Id, Name FROM TName1
GO

–-Step 6
ALTER TABLE TName1
DROP CONSTRAINT [DF_Id], [DF_Name]
GO
ALTER TABLE TName1
DROP COLUMN Id, Name
GO

–-Step 7
sp_RENAME ‘TName1.City’, ‘Area’ , ‘COLUMN’
GO

–-drop that last trigger
DROP TRIGGER TR_Trigger2
GO

–-Step 5
ENABLE TRIGGER TR_Trigger ON TName1

Weakest Database Design

funkysi1701@gmail.com (funkysi1701) — Tue, 21 Apr 2015 20:00:45 +0000

I have been recently working on a database that hasn’t been designed but has been hacked together by lots of people over the years.

It started life as an Microsoft Access database created by my director to keep track of all the projects that our company does. It was not much more than one table listing everything. At some point the data and frontend was split with the data being stored in SQL Server and the front end being an Microsoft Access ADP.

Over the years more and more columns have been added to this table that it long ago became pretty much unmanageable. A few normalized tables exist but for the most part everything is stored in this one table which makes querying it very difficult. At the last count there were 81 columns in this one database table.

Attempts to improve this database have been tried before but due to the fact that the whole company basically runs off this database and no one person knows everything that the database needs to do causes any changes to be very difficult. I suspect it may be beyond my abilities but I am going to give it a go.

One advantage I have is version control. All the tables and the structure is in version control, so when I break things I can roll them back. Any changes to the front end are also in source control, these are harder to roll back in case of problems but not impossible.

I have met with department heads to try and establish which of the 81 columns are being used and which could be dropped. Database design shouldn’t be a democracy, your databases should be designed in a way to store your data most efficiently. In this case it felt a bit like the database was playing “the weakest link” with everyone voting off unpopular columns. “You are the weakest database column. Goodbye!” After much discussion I have over 30 columns that can be dropped and 10 or so that require further investigation.

Using the Visual Studio project of the database, I removed the columns. This generated thousands of errors as most columns are included in loads of views and stored procedures. All these references need removing, so not a small job. This is made worse by the fact that the views and stored procedures are in a mess, and it is very hard to read the T-SQL code and see what is doing what. Part of my next job is going to be tidying up all these queries so I can see what’s what.

The advantage of using the Visual Studio project to do these changes is that I can generate a SQL script of the changes and I can rerun a few times and test restoring different columns as it is unlikely to be a smooth deployment.

Raspberry Pi security camera project

funkysi1701@gmail.com (funkysi1701) — Wed, 15 Apr 2015 20:00:45 +0000

So my first Raspberry Pi project is a security camera. If you have read my previous Raspberry Pi posts you will know that I have been playing about with the camera but I had one or two problems with Pidora so I am using Raspbian for this example.

To take a photo with the camera module you can use the command raspistill, this has lots of options you can view the full list by just running the command with no arguments. I will list a few options I have used.

-o filename : this outputs an image to a file with name filename -vf : vertical flip -hf: horizontal flip these are useful if the camera is upside down etc -q number: quality of jpg image between 0 and 100, the default image size is around 3 Mb, I reduced it to quality = 10 which gave me an image size of 300Kb which is more reasonable -a text: annotate you image with some text, I found that the text needed to start with a character, numbers only failed for me.

I want to script the creation of images, so I decided to name the files my script created with the time and date.

This is fairly easy with a bash script.

DATE=$(date +”%Y-%m-%d_%H%M”)
raspistill -q 10 -vf -hf -a a$DATE -o /var/www/$DATE.jpg

The first $DATE is the date I have added to the image itself, because of the problem I encountered with annotating numbers only I have added an extra a to the start of this.

It is now very easy to get CRON to run this script every few minutes, I decided to run it every 2 minutes.

My next script tidies up so you don’t get overrun with images.

cd /var/www
ls *.jpg > stills.txt
mencoder -nosound -ovc lavc -lavcopts vcodec=mpeg4:aspect=16/9:vbitrate=8000000 -vf scale=1920:1080 -o timelapse.avi -mf type=jpeg:fps=24 mf://@stills.txt
rm /var/www/*.jpg

The third line in this script does something really rather clever it creates a video from all the images you have taken. In order for this to work you will need to install mencoder (sudo apt-get install mencoder)

This is all rather good but wouldn’t it be rather cool if this images were uploaded somewhere. If someone was up to no good and they might want to disable your security camera. My first thought was to use something like DropBox, but unfortunately it is not supported by the processor on a raspberry pi. However I have found copy.com to be rather good for syncing files.

To install just run wget https://copy.com/install/linux/Copy.tgz and extract the file to install the client on your raspberry pi.

./CopyConsole -daemon -u=gary@example.com -p=’mypass’ -root=/home/pi/Copy

This command syncs the folder you have set in -root with copy.com.

The last thing I did was create a php file running on the webserver that allows you to scroll through the different images. I experimented with using javascript to auto scroll through the images but I found the manual way more useful. All the files I have written I have added to my GitHub page.

The Raspberry Pi Adventure Starts

funkysi1701@gmail.com (funkysi1701) — Sat, 11 Apr 2015 20:00:45 +0000

I am currently buzzing with excitement about my Raspberry Pi. I will try to document here what I have done so far.

I decided to buy the camera module along with my Pi. It was straightforward to connect them up, lift the connector behind the network port, insert the ribbon cable, push connector back down.

Next I connected up Keyboard, Mouse, Monitor, SD Card and Power. And my Pi had life.

I was then given a screen full of options to install, Pidora is an OS which is based on Fedora which I have some experience of so I went with that one.

After that was installed I was presented with a standard desktop with various programs installed.

So far I had not plugged in a network cable because that would mean sitting in the hall next to the router, I thought before I do that lets see if wifi will work. I have a usb wifi adapter that I use with my PC, so I plugged that in.

Nothing happened, having some experience of linux I wasn’t surprised. I went to network connections filled in the name of my wifi connection and its password and it connected. WOW! That was easy.

So I then installed webmin so I could remotely administer it and updated it with YUM. I also started the SSH service so I can use putty to login to my Pi from the comfort of my laptop. Even better than that I found a SSH client for my smart phone so I can even control my Pi from my phone!

EDIT: I forgot that in order to get webmin to work I needed to do the following.

mv /etc/redhat-release /etc/redhat-release.bak echo “Fedora release 20 (Heisenbug)” > /etc/redhat-release

OK So now I am able to configure settings on my Pi from a browser or use SSH (Secure SHell) to login to run any command I like.

Now lets lookup how to use the camera. The documentation says that the camera first needs enabling using the command raspi-config. This command doesn’t exist in Pidora. Lets skip it and see if it works anyway. It does! The command raspistill -v -o test.jpg took a picture of my ceiling.

Now how do I look at this picture remotely. SSH is a command line only interface, so I can see a 2Mb file has been created. How about installing a web server on my Pi?

yum install httpd

This install apache which is the default (and very popular) web server.

service httpd start

This starts the web server. Going to https://[IP] where [IP] is the ip address of your raspberry pi will show you the default apache page.

Now if you run raspistill -v -o /var/www/html/image1.jpg it will create your photo on the web server.

Not bad for my first steps, the command above could be scheduled and images taken every hour. There is also options for recording video.

Building something with Raspberry Pi

funkysi1701@gmail.com (funkysi1701) — Sun, 05 Apr 2015 20:00:45 +0000

Many years ago I spent a lot of time learning about electronics. It was great fun, I can’t remember specific things I built, but learning what resistors and capacitors do was fascinating.

I haven’t really thought about this much as my interests have been concentrated on computers and programming, but recently my interest has been peaked again.

The CodeNewbie and Hanselminutes podcasts have been running a month of stuff about making things and doing clever things with hardware called #Marchisformakers , this began to start me thinking about this stuff again.

Yesterday a friend of mine was showing me what he had been doing with a Raspberry Pi and Arduino. It is amazing what you can do with these tiny bits of technology and I was very impressed.

Now I am thinking what mini project could I play around with and which piece of hardware should I base it on. I don’t know much about the two platforms yet but I have found a blog post that introduces the two. My current thoughts are to start with the Raspberry Pi, it is a UK technology and it uses a linux OS (which I have some knowledge of).

Being a windows developer, most of my programming knowledge is with C# and Visual Studio. To use my existing knowledge I could install and use MONO, which means I can program the Raspberry Pi using C# (see https://logicalgenetics.com/raspberry-pi-and-mono-hello-world for more details about MONO on Raspberry Pi).

But deciding on a language to use, doesn’t help me choose a project. I haven’t even got myself a Raspberry Pi yet so I may decide more once I have had a play around with it. Thoughts so far include getting wi-fi to work on the Raspberry Pi or maybe adding a camera and sticking it somewhere and taking pictures, maybe even build myself a baby monitor (might go down well with the wife if its baby related)

Building a CI Server with TeamCity

funkysi1701@gmail.com (funkysi1701) — Wed, 01 Apr 2015 20:00:45 +0000

TeamCity is a Continuous Integration software package which is very easy to install and set up. So easy I managed it over a weekend.

First of all I fired up a VM in Azure. I found that a Basic A1 VM was sufficient so running a CI server doesn’t have to be expensive, alternatively you could always set a Virtual or Physical server on your own network, but that of course takes a bit longer to set up.

Before I get too far you are probably asking what is Continuous Integration. A Continuous Integration server takes your code from your source control repository, it then automatically builds it, runs automated tests against it and can even deploy it to production. The main advantages of this is that you can see the health of you code, you can also test different configurations.

So once you have a server ready, you need to install a few things. First TeamCity , just install with the default settings. My project is a Visual Studio project so I also needed to install Visual Studio (They have a free edition which does everything I need now, Thank you Microsoft) and SQL Server Express Edition.

Once TeamCity is installed the rest of the config can be made via the web interface. To make it easier I pointed a domain at my VM (a CNAME record is probably better as an Azure IP address may change) and opened port 80 in the Azure config. First create a build config and then create a VCS Root, this is where you specify where your source control is. TeamCity supports Github and bitbucket and probably many others. You can specify which branch you want to checkout so you can test out the latest features you have written.

Next you need to configure the steps needed to build your project, whatever you do in Visual Studio can be configured here. You then need to configure how often the build should run, it can be scheduled at specific times, this is how I remember CI servers being configured but now you can also have builds being triggered after commits to source control, very clever.

That is just about all I have configured so far. My project has a few NUnit tests written for it, these are scheduled to run after every build. In my case 94 of them are currently failing so I now need to learn how my tests work.

There are additional deployment steps that can be configured but I have not investigated these yet. Notifications about what CI is doing is also possible usually by email but there are other options available.