Tuesday, October 1, 2019

Counter overflows and clock drift bugs in aircraft and missile defense systems

A counter overflow bug happens when an unsigned integer variable storing a counter reaches its maximum value; when its already at the maximum, as soon as one more value gets added to it, the variable will reset back to 0 and continue counting up from there. If the rest of the software is not expecting this counter reset, it can result in system failures. So the longer the system is running for, the nearer the system will get to a counter overflow event.

Clock drift bugs happen when an internal timer/clock gets calculated incorrectly, causing it to slowly drift out of sync with real time. The longer the system is running, the more the clock drifts and the bigger the error gets. When real time clocks are out of sync, all sorts of downstream calculations can become inaccurate.

Both of these types of bugs typically have a workaround that requires the operator to reboot the system every X hours or days.

Both counter overflows and clock drift bugs can occur in production systems when system testing didn't include a "soak test" - where you keep the system running for a very long period, typically measured in days rather than hours. The failure to pick up the bug during system testing means that a production system gets deployed with a bug, and the only workaround becomes a regular reboot of the system.

The kicker - these type of bugs have been happening in safety critical systems for at least the past 28 years

One of the most publicized cases was the failure of a Patriot missile defense system in 1991, in which the system failed to track an incoming Scud missile, due to a drifting and out of sync clock. Tragically, this particular failure resulted in the death of 28 U.S. soldiers based at a U.S. barracks near to Dhahran, Saudi Arabia. The workaround for this bug was to reboot the system every 8 hours, but unfortunately this had not been communicated to the base in time to avoid the disaster.

Another well known case was the counter overflow in the Boeing 787 Dreamliner firmware (2015), which meant the aircraft had to be rebooted once every 248 days to avoid a counter overflow bug.

The most recent case was an internal timer bug in the Airbus A350 firmware (2019), which requires that the aircraft be rebooted once every 149 hours. 

Avoiding bugs in safety critical systems

Avionics systems are among the most complex things ever built by man. There's one leading example of a successful avionics project which was built from scratch targeting high quality and zero defects. Its the Boeing 777 - Boeing's first fly-by-wire aircraft.

For the 777, Boeing decided early on to standardize on the Ada programming language, at a time when using C was the norm. Compilers for Ada had been certified as correct, and the language itself included several safety features, such as design by contract and extremely strong typing.

Ada itself came about as a way to standardize all of the programming languages in use by the U.S. Department of Defense - before Ada they were using some 450 different programming languages. Ada was originally designed to be used for embedded and real time systems.

Ronald Ostrowski, Boeing's director of Engineering, claimed that the Boeing 777 was the most tested airplane of its time. For more than 12 months before its maiden flight, Boeing tested the 777's avionics and flight-control systems continuously - 24hrs - 7days - in laboratories simulating millions of flights. 

The 777 first entered commercial service with United Airlines on 7th June, 1995. It has since received more orders than any other wide-body airliner. The 777 is one of Boeing's best-selling models; by 2018 it had become the most-produced Boeing wide-body jet, surpassing the legendary Boeing 747.

Further reading - Boeing flies on 99% Ada

Follow @dodgy_coder

Saturday, September 28, 2019

Review of Shopify - from a developer's point of view

I setup this online shop over a couple of weekends - Chow Slow - Slow Feeder Dog Bowls. After launch I continued to make various improvements and optimisations to the presentation and content of the site. Since launch three months ago, the online shop has so far taken 72 orders.

The eCommerce SaaS product Shopify was used to build the site, on the Basic Shopify (monthly) plan.

Here below are my opinions on Shopify:

Shopify - Good points

  • Many payment options - I chose the Shopify Payments option. It accepts Apple Pay and Google Pay when on a mobile browser for example. You can tell that Shopify have extensive experience with taking payments - everything is really polished.
  • Test mode works well for putting through test credit card orders and seeing the customer experience end to end. Can also password protect the site while its under development.
  • The free themes are good and can be customised for most online shops. I used the free "Minimal" theme and then customised the layout of the home page, added several pages and added a blog.
  • Great optimisations and layouts which work well on mobile. More than 60% of the traffic for the site comes from a smartphone browser as opposed to a traditional desktop browser.
  • Really useful "Timeline" admin feature on the customer and order admin pages - see all the interactions that have happened with a customer in a simple timeline display, with a timestamp shown on each interaction.
  • Excellent "Conversion Summary" admin feature - see the site behaviour of your customer from when they first visited your site to when they bought something.
  • Comprehensive "Fraud Analysis" admin feature - for orders paid by Credit Card - recommendation of whether the transaction is likely fraudulent or not: probability shown as low, medium, or high.
  • Ability for customers to sign up with either their mobile phone number or email address. Shopify automatically sends communications to the customer using either mobile SMS or by email.
  • Easy to add social media accounts, which then automatically appear on the site footer.
  • Products are easy to configure and easy to change in terms of page title and content.
  • Ability to edit raw HTML in most places.
  • Easy to setup a custom domain, with TLS/SSL included (which just works).
  • There are SEO optimisations and settings available at a page level. You can change the page title, meta description and URL.
  • Integration with Google Analytics is excellent - the 'Advanced eCommerce' option should be turned on - it results in a data going straight into GA for your conversions and sales.
  • Page load time is good - bulk of the home page loads typically in 2-3 seconds.
  • A comprehensive API for writing custom integrations with other backend software and systems.
Shopify - Bad points
  • Marketing integrations - Facebook Ads for example - difficult and error prone to setup. Facebook Ads never worked for me after many headaches trying to setup. The marketing integrations seem like a quick way to lose money fast.
  • Inability to structure the Products, Pages and Blog sections to be within a custom sub-folders structure (as is typically recommended to silo a website for SEO purposes).
  • Checkout page customization is limited to Shopify Plus users.
  • Checkout script editor is limited to Shopify Plus, so any custom logic to apply discounts has to be done through plugins.
  • API is rate limited - will be fine for most users however ERP systems which run via batch processing or via polling may have issues. You need to implement everything as event driven webhooks rather than by polling the Shopify API.
Shopify - Areas for improvement
  • I'm not convinced of the need for many of the Shopify apps (plug-ins) that get automatically suggested. I have a feeling that they will result in bigger page sizes and slower load times. I have not needed to use any apps yet.
  • Its sometimes hard to find the relevant place for a setting - whether under preferences of the online shop or under settings of the account itself. I should probably make more use of the search bar at the top for this in the future.
Shopify - Conclusion

I highly recommend Shopify for setting up an online shop. When launching, your focus will be taken up with gaining your first customers, online marketing and the delivery process, so having a solid full featured website behind you is crucial.

A bit more about our online shop - Chow Slow - Slow Feeder Dog Bowls

I setup the site for my wife, who runs the business day to day. She handles the processing of orders and deliveries. We came up with the idea of selling to this niche market after finding out about this type of product in 2018. Our pet beagle (Ripley) had an issue of regular regurgitation due to eating too quickly. The slow feeder bowl product solved his problem and results in more relaxed meal times every day.

Follow @dodgy_coder

Thursday, April 18, 2019

Inactive/Failing Mobile Apps: The Many Benefits of Regular App Updates

This article applies mainly to mobile apps which are not in active development. It applies equally to mobile apps on the public app stores and also to enterprise mobile apps which get distributed via other means (such as MDM software).

Example scenario of an inactive/failing app:
  • The active marketing of your app that happened at launch has now stopped
  • Your app's last update was several months ago
  • The uninstallation rate has started rising above your installation rate month on month

A way to prevent this happening is to release regular app updates, even when there are no major changes to your app's code.

Benefits of releasing regular app updates:

  • Gain new users; many users look at the last release date on the app store page before downloading your app. Anything over a year old can be immediately rejected as out of date.
  • Prevent uninstalls; seeing your app has been updated on their phone keeps it in the user's mindshare, so they know you are still actively developing it.
  • Update the dependencies; you will likely have several dependencies on open source libraries and mobile SDKs. The more often you release, the more often these can be updated too.
  • Update the toolchain; Xcode / Android Studio / Visual Studio get updated once every month or two. Same applies to hybrid and cross platform toolchains such as Unity, Ionic and Xamarin.
  • Update the store listing; since you're doing a build, take the opportunity to review and tweak the store listing and screenshots. Apple's App Store Connect website and Android's Google Play Console get updated regularly with enhancements you can take advantage of.
  • Less build issues; there will be a lower chance of a major build issue if you regularly update your toolchain and its dependencies. In my experience, the longer you wait between app builds and releases, the greater the chance of a major build issue the next time.
  • Source control change visibility; the project is kept active in terms of source control history; especially relevant if its an open source project with other potential developers able to see the change history on github or bitbucket.
  • Developer benefits; you don't end up with a legacy mobile app that noone can get building anymore.

What if there are no significant changes to your code?
  • Have the dependencies or toolchain been updated? If yes, that means your app has gained some potential bug fixes and enhancements for free - just by doing a build.
  • There's always some minor refactoring you can do to make small improvements to your codebase. Just spend a couple of hours if that's all you can spare.
  • The release note; just use the standard "Bug fixes, performance and stability improvements" if you can't think of anything else.

How often to release?
  • The more often the better.
  • Aim for a regular release cycle of between 1-3 months.

Further reading

Distribution of Android apps per download range

Download distribution of Android apps, and lifetime growth rates per download range

Statistics about the release schedule of apps on the Apple app store

Best practices for Apple app store updates

Follow @dodgy_coder

Subscribe to posts via RSS

Sunday, September 18, 2016

Lightweight IoT Command and Control

Many hobbyist IoT projects running on the Raspberry Pi (RPi) require a webserver running on the RPi which receives requests from a browser, where the client using the browser is connected either to the local WiFi network, or possibly (via a port forwarding setup), connected to the public Internet.

An alternative that is easy to setup, lightweight and low cost is one that makes use of Amazon's SQS (Simple Queue Service) to setup two queues. One queue is for commands - from a mobile application to the RPi. Another queue is for responses - from the RPi back to the mobile application. It results in a simple and easy to secure solution available over the public Internet. It requires no port forwarding - both endpoints make use of a client connection to Amazon AWS. Given the small amount of messages needed for typical hobbyist project, the SQS component will be covered by the AWS free tier.

Here's the high level design of the system ...

Lightweight IoT Command and Control
On the Raspberry Pi, a lightweight application is running that makes use of the excellent Python boto3 AWS SDK library. It is always waiting for a command to arrive in the command queue. As soon as it receives a command, it processes it and adds a response to the response queue.

On the mobile device an Android application is running that makes use of the AWS Mobile SDK for Android. When the user presses a command button, such as 'Open Garage Door', it adds a command to the command queue, and asynchronously waits for a response to arrive in the response queue. When it receives a response (e.g. the command's success or failure status), it updates the app's user interface.

Saturday, January 2, 2016

Custom domain for Azure Web Apps using FreeDNS

The below steps go through setting up a custom domain name on your Azure Web App, using the FreeDNS service from NameCheap. Both the www subdomain www.yourdomainhere.com and the naked yourdomainhere.com will be setup.

I have an Azure S1 Small Instance App Service Plan on which I'm running 4 web apps. Originally I had planned to use Azure DNS, to keep everything within the same cloud service, but after some initial attempts formed the opinion that Azure DNS isn't mature enough yet to host production web sites. Azure DNS can currently only be administered from within Azure Powershell - that in itself was enough to put me off using it. If you need to make a quick change to a DNS record, you don't want to be having to run Powershell to do it. Microsoft are working on adding Azure DNS to the Azure Portal, but at the moment it's not available.

I have a couple of domains through NameCheap and for those and a couple of others (with CrazyDomains) I decided to use NameCheap's DNS service.  Even when your domain is not registered through NameCheap, they still offer a free DNS service, called FreeDNS.

Note that in the below steps, yourwebappname and yourdomainhere.com should of course be replaced with your specific names.

Step 1. Reduce the TTLs to 5 minutes

Note: you only need to perform this step if you're porting from an existing DNS provider. No need to do this if you're setting up a new web app on a new domain.

Within your current DNS service, setup all the TTL settings (Time To Live) on the DNS records to the minimum (typically 300 seconds, or 5 minutes). This may help in having the DNS changes propagate faster.

After this is done, try to get a full DNS zone file listing. When you port the DNS records to FreeDNS it will help ensure that you don't miss anything. My previous VPS and DNS provider was Linode and I was porting a few web apps from Linode over to Azure web apps.

Step 2. Ensure your Azure web app is running

Ensure your web app is published and running ok at http://yourwebappname.azurewebsites.net

Step 3. Find the IPv4 address of your Azure web app

From a windows cmd.exe prompt ...

C:\> nslookup yourwebappname.azurewebsites.net

Server:  UnKnown

Non-authoritative answer:
Name:    waws-prod-xxx-xxx.cloudapp.net

The nslookup command will provide you with your web app's IPv4 address. This is shown in the second Address field above, e.g. I am running an Azure S1 Small Instance App Service Plan. I believe this method still applies to other plans including shared instance plans.

Note that if your web app is restarted or stopped this may result in it being assigned a different IP address. Be aware of this when using naked domain URLs (without the www prefix) since these rely on the A record being configured for the correct IP address of your web app (as per step 6).

Step 4. Setup FreeDNS

On the NameCheap FreeDNS page, enter yourdomainhere.com and click 'Get DNS'. If its eligible, then click 'Add to Cart', then click 'Set up DNS'.

You get this message from NameCheap :-

N.B. don't go ahead and set the nameservers just yet, that step will come later.

    Congratulations! Your domain/ sub-domain is added to our DNS service.
    Please set the nameservers of your domain/ sub-domain to


    Our system will periodically monitor your domain's DNS setting and will activate your domain once it is pointing to our servers.


Step 5. Verify ownership of your domain

In NameCheap, on the Domain List page, once the yourdomainhere.com is listed as active, then need to click on "Authorize DNS" link...

Select the required email, such as admin@yourdomainhere.com to use as the auth email.

On the domain page, the redirect domain setting will show "Your FreeDNS domain is waiting for Authorization or Verification by domain owner."

In the received email, click on the embedded hyperlink and then on the webpage, click the "I AUTHORIZE" hyperlink.

A message should display "Host has been successfully activated".

Step 6. Setup the FreeDNS records

At this step you'll setup all of the DNS records for your domain.
For all of these I left them with the default TTL setting of 'Automatic'.
In NameCheap, on the Domain List page, click the MANAGE button next to your domain, then click on the Advanced DNS tab.

In the Host records, add the A record with the IP that was returned in step 3.

Type  Host  Value            TTL
A     @          Automatic

Add any required TXT records (often used for verification).

Add the 3 CNAME records required to verify ownership of the domain to Azure.

1. Set HOST to awverify
    Set Target to awverify.yourwebappname.azurewebsites.net

2. Set HOST to awverify.www
    Set Target to awverify.yourwebappname.azurewebsites.net

3. Set HOST to www
    Set Target to yourwebappname.azurewebsites.net
If required, set the mail setting to Custom MX and add the MX server records...
For example if you have Gmail enabled on the domain you'd use these MX records...

Set Mail to Custom MX

Host        Type Priority      Mail server name
@            MX     20         ALT1.ASPMX.L.GOOGLE.COM.
@            MX     10         ASPMX.L.GOOGLE.COM.
@            MX     20         ALT2.ASPMX.L.GOOGLE.COM.
@            MX     30         ASPMX2.GOOGLEMAIL.COM.
@            MX     30         ASPMX3.GOOGLEMAIL.COM.

Step 7. Change the DNS nameservers using your registrar

For the case when your registrar is not NameCheap, you'll need to point the nameservers for the domain to the FreeDNS nameservers.

Log into your registrar's website and set the following nameservers, removing the existing nameservers if required ...


Step 8. Wait for the DNS nameserver change to propagate

You can check whether the changes have propagated yet using nslookup from the windows command line ...

C:\> nslookup yourdomainhere.com
C:\> nslookup www.
C:\> nslookup awverify.
C:\> nslookup awverify.www.

Use nslookup in interactive mode for MX and TXT record lookups ...

C:\> nslookup

set q=mx


set q=txt



Or you can also use these great online tools for checking the DNS propagation ...


Or on Linux I believe the command is: dig yourdomainhere.com

Step 9. Bring the domains into your web app using the Azure Portal

In the Azure portal, go to yourwebappname Web App -> Settings -> Custom Domains and SSL -> Bring External Domains

Enter yourdomainhere.com into the field, and then press tab. Wait for it to verify your CNAME records.

In the next field, enter www.yourdomainhere.com and press tab again. Wait for it to verify your CNAME records.

It should succeed as you tab out of each field, and after you click the save button at the top, it should report "Updating hostname bindings".

Step 10. Test in a browser

Open both www.yourdomainhere.com and yourdomainhere.com in a browser and confirm that they load ok.

Turn on the browser's web developer tools (Network tab) to check that all the components of the web app are loading with HTTP 200 OK status, and are resolving to the new IP address.

Its normal to experience some issues for the first several hours after changing nameservers, for example you may see the domain resolve to the old name server settings or flipping back and forth from new to old. Usually this will settle down after a few hours. It can be caused by the browser caching DNS results, something I've noticed particularly bad in Firefox sometimes. You may also need to flush the DNS cache on the computer you are using.

Flush the DNS cache in windows via:

C:\> ipconfig /flushdns

Other references ...

There is some additional Azure related information in the below post about this subject:

Follow @dodgy_coder

Subscribe to posts via RSS

Sunday, December 13, 2015

Low cost airlines online pricing hacks

Low cost airlines are the masters of dynamic pricing, bundling and pricing psychology.

Every customer who makes an online booking presents them with some new data to plug into their dynamic pricing algorithm. They also analyse the best way to present the various upgrades available to purchase before the final checkout, to maximise the customer's spend.

I've just taken a look at budget carrier JetStar's booking experience and its use of various pricing techniques and psychology.

After you've searched and selected your flights you'll be presented with a choice. Look at how unattractive they make the "Starter fare" default option ...

The "Plus bundle" at $106 is the one JetStar wants you to pick. The "Max bundle" at $753.49 is just there as an anchor price, to give you the impression that the middle one is good value. The "Starter fare" is the one they want you to avoid. They do this through the selective use of negative and positive language and the colours of grey, green and blue.

The Starter fare

Comfortable leather seat
This is patronising ... is there so little going for this bundle option that they have to mention that you actually get a seat, as opposed to standing in the aisle?

7kg carry on baggage
Strict size and weight limits apply
Negative tone. 7kg is not enough for most people's needs.

Checked baggage not included
Repeated, negative tone.

Starter fares are non-refundable
Negative tone.

... compare this to the most popular option, the option JetStar wants you to choose ...

The Plus bundle

No change fees

2,225 Qantas points
Free money. Only $30 credit off your next fare, but its better than nothing.

Free standard seat selection
The power of free.

More included stuff.

20kg checked baggage
Notice they don't mention that a strict size and weight limit also applies to the 20kg baggage.

People are scared of change fees

Most people when seeing the starter fare option will get the impression that if they need to change their flight for any reason, perhaps due to a flight delay, they are in trouble. In fact, JetStar covers you for flight delays and will make the necessary arrangements and additional bookings free of charge. But they only mention this in the fine print.

Most people want checked baggage

You also get the impression that you have to choose the plus bundle or max bundle if you want checked baggage. Actually, you can choose the starter fare, and then later on in the process you can optionally pay for only the specific amount of checked baggage that you need, at a rate of $46 for 15kg, $50 for 20kg and $62 for 25kg. Most customers likely don't know that these cheaper options are available, and will just choose the plus bundle, paying $106, when all they needed was checked baggage. You can select the Starter fare, then pay separately for 20kg checked baggage ($50), a meal ($15) and a reserved seat ($6) and the total will be just $71, a saving of $35 on the Plus bundle.

Noone wants to be randomly assigned a seat

When it comes to selecting a seat, its a bit disingenuous for JetStar to claim this ...

"You have not selected a seat. If you do not wish to pre-select a seat one will be assigned for you randomly from what is still available at check-in."

When you check in at many airports, assuming you get to the airport early and the plane isn't sold out, you'll still be able to select what type of seat you want - the check-in staff are usually more than helpful and should be able to provide your choice of a window or an aisle seat.

If you're a family with children, the airline will try to assign an adult family member to be sitting close to each child in your family, even when you didn't pay to reserve a seat. If you can't be accommodated at the point of check in, you would likely be able to work something out after boarding by speaking to the cabin crew.

Also, remember this inconspicuous bit of writing at the bottom ...

"Jetstar will attempt to accommodate your seat preference, however due to operational considerations cannot guarantee that your seat allocation will be as your selected preference."

So in other words, they ask you to pay for a seat preference, but cannot guarantee it.

Follow @dodgy_coder

Subscribe to posts via RSS

Monday, June 15, 2015

The Default Files Folder Pattern

Software installation best practices

When you've got critical settings files that absolutely should never be overwritten or cleared back to defaults, this pattern has always worked well for me. It can apply to both Windows, Linux and Mac based software.

The installation method can either be a simple file copy utility/archive or something more complex like a Windows installer.

An example target structure might look something like this:

    /bin           => Application binaries.
    /defaults      => A default version of all
settings files.
    /docs          => Documents or manuals.
    /settings      => Critical settings files.

The settings folder contains the critical settings files and is not part of the installer. The installer includes only the bin, defaults and docs folders. So installing the software only overwrites bin, defaults and docs, but never the settings. This applies equally to Windows installer packages - for this case, just don't include the settings folder in the Windows installer package. The defaults folder contains the default version of all of the the files expected in the settings folder.

Whenever the software accesses a particular file in settings the following lazy initialisation logic is used ...

- Does the settings folder exist?
    - Yes: Continue
    - No:  Create the folder and continue
- Does the file exist?
    - Yes: Open it
    - No:  Copy the file from the
defaults folder into the
           settings folder, then open it

By following the above logic, you ensure that an existing critical settings file is never overwritten.

From reading the latest report on the deadly Airbus A400M crash, its claimed that the cause was that their software installation procedure did the equivalent of overwriting a settings file or clearing it back to defaults. In this case, the settings file contained calibration parameters that were critical to the operation of 3 out of the 4 propeller engines.

Follow @dodgy_coder

Subscribe to posts via RSS