Personal Blogs

I decided that I had to go back to the MongoDB part as, although I was now conversant with Riak, I didn't know enough about MongoDB to tackle the assignments. In the process, I installed on my Mac (using Brew) and found, to my surprise, that it worked first time! I was then able to play with the code in the "Definitive Guide" book. I then went back to the Notebooks and got confused - again - as it appears that this uses its own MongoDB database. I was expecting it to use the one I installed if it was there listening on the right port. I forgot that the Notebook is running inside Vagrant so couldn't see the other instance. Having messed about like this, I decided to finish off the ICMA - which contains questions that require coding to answer!

I answered all of the usual questions and got 29% - I need 30% in 5 ICMAs so if I could get a few more marks, I could put this to bed and stop worrying about ICMAs for the rest of the course. I went through the coding questions and came to one where they were looking for the month with the least accidents. I guessed a month and found that it was right! 10 marks! I ended up with 51.66% so quickly submitted it and now I am ICMA free.

I have now completed Question 1 of TMA 02 so I have plenty of headroom for the rest of the assignment as it isn't due until 16th June.

To bore you with more of my Riak activity, I have been using my son-in-law's expertise to play with the interface from my Smalltalk code into the database. Riak has an HTTP interface, which is what I am using. It also has a direct socket access which requires converting all the data into hex and so on. It has one advantage in that, once connected, Riak keeps the socket open. This should, in theory, provide a much faster interface. However, I must be doing something wrong because, as we now have the GET operation coded using both HTTP and the Protocol Buffers, I did some timings and found that my HTTP interface was almost twice as fast as the direct socket version. I think that we have to examine what our socket stuff is doing as it should be the other way round.

Lastly, I was having problems using JSON as the data format - the JSON interface that I was using comes from the Squeak world and I had to recode it to run under my VisualAge Smalltalk. In the end, I decided that the idea that Riak is a JSON database is a bit of a misnomer as it will accept anything that you give it. I decided to rework the data formats and save all of my objects in CSV format. I already had all the code to do this as it was CSV based before I started with Riak. Anyway, it is all converted and my software is running perfectly! I can now put the Riak stuff to bed for now and, hopefully, pick it up again when I propose my project at the end of the degree (12 months away).

I have not found this third week particularly easy. I found that Notebook 04.5 particularly time consuming. I am not sure quite where the authors of the course got their timings from but I thought that the following was a bit optimistic:

"Activity 4.5 Notebook - 30 minutes"! At the end, it presents you with a list of tasks:

• a) Show a count of the number of sales records for each District.
• b) Show a count of the number of sales records for each Team in each District, including the Team and District margin totals.
• c) Show the total sales value for each Team in each District summed over the year.
• d) Show the total sales value for each Team Member in each District over the year, showing the District and Team member margin totals. (Remember you need the team name and salesperson name to identify each person uniquely.)
• e) Show a bar chart of the number of sales each month. 
• f) Show a bar chart of the total sales each month.
• g) Show a scatter plot showing the Item Cost v. the number of Units in each record.
• h) Add a Season column to the DataFrame. For each sale record, the value for Season will be derived from the month: (11,12,1) are Winter, (2,3,4) are Spring, (5,6,7) are Summer, (8,9,10) are Autumn. From the sales in each Season calculate the number, average, maximum, minimum and total sale amount over the season (that is, from all the sales records grouped by season report the number of records, and the average, maximum, minimum and total sales amounts).

These would prove to take me 1 1/2 hours on their own and I never did finish so quite where the 30 minutes for the whole notebook came from, I have no idea. Perhaps one of the course authors timed themselves? Anyway, I think that you would need to be very adept at Python and pandas to achieve the time. 

I left that notebook and tackled the next one - about Regular Expressions. I found this not quite so difficult but it did require some careful thinking. By this time, I was getting quite depressed about my chances of even getting to do TMA01, let alone complete it and continue. Fortunately, there was a post on the forum which set me back on the path of confidence (well - a bit). Someone mentioned that there wasn't an exam at the end so we had no need to have everything welded into our brains and that we could manage throughout by referring back to the course notes. A sign of understanding and relief was heard in my household.

On that basis, I tackled the first part of TMA01 and found that I could manage quite well! So, onward and upwards, as the saying goes.

I have been writing successful software for the last 35 years - some of it in mission critical situations such as bank currency trading rooms and so on. This has been done in Basic, which was taught to me by the OU in M251 An algorithmic approach to computing (in 1977), in UCSD Pascal (which was the OU's next teaching language and, finally, Smalltalk - which was the OU's third teaching language (replaced by Java and now Python to some extent). In all that time I have relied heavily on having documentation to hand as I have never managed to learn by rote all of the syntax of these languages. As an aside, Smalltalk (like pandas) has a very easy syntax but an extremely rich library so I spend some time regularly looking up stuff.

The upshot is, I think, that I should assume that I will do the same with Python, pandas and Regex and not try and have it all in my brain. I have one day left to complete week 4 to maintain my one week buffer so it is eyes down tomorrow for a good stint.

One last comment. Last week, I signed up for TM354 - Software Engineering, which starts in October. I have to say that those who know me from my development background expressed opinions that this would be quite boring given my CV. I felt that it would be a reasonable step to the final project course (TM470). However, it turns out that it has an examination. Ignore what I said above as I would be quite happy to sit this if it wasn't for my stupid arthritis and the problems it brings with continuity of concentration. As a result, I cancelled the application and signed up for TM352 - Web, mobile and cloud technologies. Firstly, this doesn't have an exam component but also it ties in very nicely with my thoughts on a project for TM470. More on this later.

Getting closer meant that I had to get more serious about learning Python. 

I run a web and e-mail server from home. The e-mail server is for the family domain and there are three web sites that are running on the machine as well. I thought that I had put together a reliable setup when I  purchased a brand new Lenovo desktop box running Windows 8 (which was fairly quickly upgraded to Windows 10). However, the box freezes occasionally. I can't find any reason for this so I just have to manage the situation. I find the the Pingdom service, which is supposed to e-mail me when my server goes down isn't as reliable as I need.

I was casting around for another program to write in Python to extend my knowledge of the language so I thought that I would write my own Ping software. Now a ping is something that gets sent to a web address and reports back if the server at that address responds or not. What my software had to do was to regularly connect to the server and, if the server failed to respond, send me an e-mail. There was a few problems with me doing this in Python.

1. I had no idea how to connect to a web server
2. I had no idea how to send an e-mail
3. I had no idea how to log these actions

So, this was going to test my ability to learn the inner details of Python. It was actually easier than I thought. Python has a rich set of libraries so I included the following in my program:

import smtplib
import requests
import time
import configparser
import os

My next progression towards the course is a giant backward step. We have decided to move apartments and the move takes place on 23rd January. We are only moving from one floor to another (the new apartment is much bigger than the existing one) but, for various reasons, we are to lose our broadband internet connection for about 10 days - which covers the date of the start of the course! I am hoping that I can use my iPhone as a router for my iMac in the meantime. As we have a nice 4G signal here, that shouldn't be a hardship.

Lastly, I have quite severe Arthritis in a lot of my body so I have registered, through the OU, as disabled. Following an assessment of my problems, I have been awarded a grant that will provide me with equipment that will help me in my studies. I have difficulty sitting for too long in one position and find concentrating for long periods a trial due to pain levels.  Yesterday, I had a visit from a representative of an Ergonomics company who has carried out an assessment for a new desk, chair and reading aids. It seems that I am to get a desk that is electrically height adjustable, a fully configurable chair, an iPad holding device and a footrest. All of this is designed to get me into the best possible position for sitting at my iMac for extended periods. It seems to take about 6 weeks for this to happen but it does mean that, soon after the course starts, I will be able to get on with more success. My existing hobby room is way to small to take the new desk along with my other stuff so it is good that I am moving to a bigger apartment as the 2nd bedroom in that one is substantially larger than now. I might even have room for a decent model railway - at last!!!

I last studied with the OU back in 1980 so all of my previous credits are from that time. I have an original E student code - which amazingly I still remember. Due to Student Loan requirements, I am unable (as explained later - also unwilling) to take the two required prerequisites for the course I have signed up for. This course is TM351 Data management and analysis. 

I studied computer subjects back in the 1970s previous to the advent of micro-computers but was prepared for them when they finally arrived around 1978. At the time I was a Foreign Exchange Dealer in the City and, up to them, had assumed that the OU course was a side show and not relevant to my daytime occupation. Having attended a one week summer school at Reading University for my T101 Mathematics course, I returned with a program to do some elementary calculations on Money Market problems. My boss liked what I had done and arranged for me to obtain one of the first Z80 micro-computers to design a suite of programs for the dealing room. This was a North Star Horizon. This, along with an Elbit VDU was used to build a good suite of software. The bank, eventually, bought an Horizon for me to have at home.

I moved jobs and arranged to keep the Horizon. From there I wrote a whole package of front office software for the bank - including a package for our branch in Greece - built in Microsoft Basic on an Apple II. Eventually, I left the City and formed my own software company - in 1985. For the next 20 years I spent my time developing software in the Smalltalk programming language. Smalltalk is the original Object Oriented language - designed at Xerox PARC in the mid-1970s.

Consequently, I saw no reason for me to study OO programming using Java (which was one of the prerequisites) as I was already very competent in this field. After me writing a CV and obtaining a reference from an insurance company in Connecticut, USA (I had been writing custom software for them for the last 20 years!) the course team agreed that I had sufficient background to take the course without the prereqs.

I will still need to get up to speed with Python. To assist this, I am currently working on the Future Learn three week course - “Learn to Code for Data Analysis” which uses Python to analyse spreadsheet data. Once this is complete, I have designed a few projects to fully understand Python - one of which is to convert my existing Smalltalk web program (using Seaside) which tests vocabulary knowledge for those learning Latin!

So. Onwards. More on the Data Analysis course and Python next time.