Dodgy Coder

Google's fiber leeching caper

2013-02-19T21:04:00.000-08:00

Back in 2000, Google only had data centers on the US west coast and were planning an expansion over to the east coast, to reduce latency to end users. At the time, Google was not hugely profitable like today, and were very conscious of costs. One of the biggest costs of the move was duplicating the data contained in their search indexes over onto the east coast. Google had just passed indexing 1 billion web pages, and had around 9 terabytes of data contained in their indexes. They calculated that even at the highest speed of 1 Gigabit per second, it would take 20 hours to transfer all the data, with a total cost of $250,000.

Larry and Sergey had a plan however, and it centered on exploiting a loophole in the common billing practice known as burstable billing, which is employed by most large bandwidth suppliers. The common practice is to take a bandwidth usage reading every 5 minutes for the whole month. At the end of the month, the top 5% of usage information is discarded, to eliminate spikes (bursts). They reasoned that if they transferred data for less than 5% of the entire month (e.g. for 30 hours), and didn't use the connection at all outside that time, they should be able to get some free bandwidth.

So for 2 nights a month, between 6pm and 6am pacific time, Google pumped the data from their west coast data center to their new east coast location. Outside of these 2 nights, the router was unplugged. At the end of the month the bill came out to be nothing.

They continued like this every month until the contract with their bandwidth supplier ended, and they were forced to negotiate a new one, which meant actually paying for their bandwidth. By this time, Google had started buying up strategically located stretches of fiber, paving the way for its own fiber network to support its increasing bandwidth needs.

Source:

In The Plex: How Google Thinks, Works, and Shapes Our Lives [Amazon]
By Steven Levy
Published: April 12, 2011
See pages 187-188, Steven Levy's interview with Urs Hölzle and Jim Reese.

Follow @dodgy_coder

Subscribe to posts via RSS

How to learn a new programming language

2013-02-01T22:29:00.000-08:00

Here's some tips on learning a new programming language. They aren't listed in any specific order. Hopefully you'll gain from this at least one new tip that will help you to become proficient in the next language you learn.

Build something you actually need right now. This could be either a tool you can use in your day job, or something useful you can make use of at home.

Consider these example projects ...
- Write a simple unit testing framework. Many new languages don't have any unit testing frameworks available when they are first introduced. This project will force you to use areas of the language like reflection and meta-programming. Once completed, it becomes useful straight away, for unit testing your future work in the language.
- Implement a disk usage tool; it summarizes the disk usage of all directories on a disk and outputs to the console. It doesn't require overly complex algorithms but touches a lot of the basics: recursion, filesystems, command line parsing and output formatting.
- Implement a backup/archive script which has command line switches to exclude certain file extensions. It should place the backup into a single .zip or .tar.gz file. The project will touch on the following: recursion, filesystems, command line parsing, compression libraries and regular expressions.
Port an existing, well known program to the new language. Since you are porting it, you already have the application design work done. This frees up your mind to focus on the specifics of the new language. After you've finished you'll then have a good reference to which you can refer back to, when comparing the old with the new language.
Find a decent book on the language and read through it all as fast as you can. The goal is not to memorize everything, but to soak up the ideas and idioms of the language. Then write some small but non-trivial project. When you get stuck, hopefully it'll trigger a memory of something from the book, and you can go back to refer to it.

Many of the "in a Nutshell" and "Head First" series of books, both published by O'Reilly, are highly rated by readers - they are available for many popular languages (C, C++, C#, Java, Python, JavaScript, PHP).
Mix action with equal parts learning (reading books/tutorials). Too much action without learning, and you get a lot of poor quality code. Too much learning without enough action, and you fail to absorb the material to a deep enough level.
Study reference works on public repositories. Find a medium sized project on GitHub which is written 100% in the language. Read though the code and try to understand it. Look for projects written by the language designers or an acknowledged expert.

As an example, with Go, the standard libraries for the language are written in Go and are open source, e.g. here is part of the strings package. In addition, Brad Fitzpatrick and other members of the Go team have several projects on GitHub that you can read and learn from, e.g. here is a Go client for Memcache.
Devote large, uninterrupted chunks of time, at least half a day, to learning the new language. Brief, half hour sessions over the course of the week aren’t really useful, because most of the time would be spent just getting back up to speed on what you previously studied.
Learning a language shouldn’t just be a solitary endeavor. There are plenty of people who have made the same mistakes that you have, so asking for help is a great way to overcome problems when you get stuck. Some possible sources of help online: Language-specific IRC channels, StackOverflow, Twitter, Facebook groups, Quora, Google+, Google groups. You can also submit your finished code to these forums after you've completed a project; people more experienced with the language than you will often be able to identify areas which can be improved or simplified.
Use an editor with syntax highlighting. Perennial favorites such as Vim and Emacs, plus newer editors such as Sublime Text, feature support for most if not all programming languages and are available for all major operating systems. Some languages are often associated with specific IDEs; these are a good idea when learning a new language. These are generally ...
- Eclipse IDE for Java and Android development.
- Xcode for Objective-C and iOS development (on Mac OSX only).
- Visual Studio IDE for C#, C++, VB.NET, F# (on Windows only).
All of the IDEs and editors listed above are either completely free or have an unlimited trial version available.
Working on a real project with real customers and deadlines is a white hot crucible for learning a new programming language. If you really need to learn a language quickly, then consider taking on a new job which requires it. Once you've got the job, you'll have no other choice but to learn it quickly.
Finally, a tip from The Pragmatic Programmer, Tip #8 "Invest Regularly in Your Knowledge Portfolio":
"Learn at least one new language every year. Different languages solve the same problems in different ways. By learning several different approaches, you can help broaden your thinking and avoid getting stuck in a rut. Additionally, learning many languages is far easier now, thanks to the wealth of freely available software on the Internet."

Follow @dodgy_coder

Subscribe to posts via RSS

Big Ball of Mud Design Pattern

2013-01-15T04:23:00.001-08:00

In "Big Ball of Mud", Brian Foote and Joseph Yoder propose that the default (and most common) software architecture in use is the "Big Ball of Mud" pattern and go on to discuss six additional patterns and activities that it gives rise to: "Throwaway Code", "Piecemeal Growth", "Keep it Working", "Shearing Layers", "Sweep it Under the Rug" and "Reconstruction".

Their original article is located here and can be downloaded in PDF form here: Big Ball of Mud by Brian Foote and Joseph Yoder [PDF download]

I have picked out what I think are the highlights of their article ...

Big Ball of Mud ... alias Shantytown or Spaghetti Code

Shantytowns are usually built from common, cheap materials with simple tools and using unskilled labor. The construction and maintenance of the shantytown is labor intensive, and there is little or no labor specialization - each builder must be a jack of all trades. There's no overall planning, or regulation of future growth.

Too many of our software systems are, architecturally, little more than shantytowns. Investment in tools and infrastructure is often inadequate and the tools that are used are primitive. Parts of the system grow unchecked, and the lack of architecture and planning allows problems in one part of the system to erode and pollute adjacent portions. Deadlines loom like monsoons, and architectural elegance seems unattainable.

The time and money to chase perfection are seldom available and there is a survival at all costs attitude, to do what it takes to get the software working and out the door on time. The biggest cost borne by the Big Ball of Mud development is the lack of a decent architecture.

Common features of Big Ball of Mud Code

Data structures are haphazardly constructed, or non-existent.
Everything talks to everything else.
Important state data is global.
State data is passed around though Byzantine back channels that circumvent the system's original structure.
Variable and function names are uninformative and misleading.
Functions use global variables extensively, as well as long lists of poorly defined parameters.
Functions themselves are lengthy and convoluted, and perform several unrelated tasks.
Code duplication.
The flow of control is hard to understand, and difficult to follow.
The programmer’s intent is next to impossible to discern.
The code is simply unreadable, and borders on indecipherable.
The code exhibits the unmistakable signs of patch after patch at the hands of multiple maintainers, each of whom barely understood the consequences of what he or she was doing.
Did we mention documentation? What documentation?

Working with Big Ball of Mud Code

Some software engineers come to regard life with the Big Ball of Mud as normal and become skilled at learning to navigate these quagmires, and guiding others through them. Over time, this symbiosis between architecture and skills can change the character of the organization itself, as swamp guides become more valuable than architects.

As per Conway's Law, architects depart in futility, while engineers who have mastered the muddy details of the system they have built, prevail. The code becomes a personal fiefdom, since the author care barely understand it anymore, and no one else can come close. Once simple repairs become all day affairs, as the code turns to mud.

Throwaway Code ... alias Quick Hack or Protoduction

While prototyping a system, you're normally unconcerned with how elegant or efficient your code is. You plan that you will only use it to prove a concept and once the prototype is done, the code will be thrown away and written properly. As the time nears to demonstrate the prototype, the temptation to load it with impressive but utterly inefficient realizations of the system’s expected eventual functionality can be hard to resist. Sometimes, this strategy can be a bit too successful. The client, rather than funding the next phase of the project, may slate the prototype itself for release.

This quick-and-dirty coding is often rationalized as being a stopgap measure. More often than not, the time is never found for this follow up work. The code languishes, while the product flourishes. It becomes a protoduction - a prototype that gets used in production.

Once it becomes evident that the throwaway code is going to be around for a while, you can turn your attention to improving its structure, either through an iterative process of Piecemeal Growth, or via a fresh draft, as discussed in the Reconstruction pattern below.

Piecemeal Growth ... alias Refactoring

Successful software attracts a wider audience, which can, in turn, place a broader range of requirements on it.

When designers are faced with a choice between building something elegant from the ground up, or undermining the architecture of the existing system to quickly address a problem, architecture usually loses.

In the software world, we deploy our most skilled, experienced people early in the lifecycle. Later on, maintenance is often relegated to junior staff, and resources can be scarce. The so-called maintenance phase is the part of the lifecycle in which the price of the fiction of master planning is really paid. It is maintenance programmers who are called upon to bear the burden of coping with the ever widening divergence between fixed designs and a continuously changing world.

Piecemeal growth can be undertaken in an opportunistic fashion, starting with the existing, living, breathing system, and working outward, a step at a time, in such a way as to not undermine the system’s viability. You enhance the program as you use it. Massive system-wide changes are avoided - instead, change is broken down into small, manageable chunks.

Keep It Working ... alias Continuous Integration

Businesses become critically dependent on their software and computing infrastructures. There may be times where taking a system down for a major overhaul can be justified, but usually, doing so is fraught with peril. Therefore, do what it takes to maintain the software and keep it going. Keep it working.

This approach can be used for both minor and major modifications. Large new subsystems might be constructed off to the side, perhaps by separate teams, and integrated with the running system in such a way as to minimize disruption.

A development build of each product can be performed at regular intervals, such as daily or even more often via an automated build tool. Another vital factor in ensuring a system's continued vitality is a commitment to continuous testing, which can be integrated into the automated build process.

Shearing Layers

Software never stands still. It is often called upon to bear the brunt of changing requirements, because, being as that it is made of bits, it can change.

Over time, the software's frameworks, abstract classes, and components come to embody what we've learned about the structure of the domains for which they are built. More enduring insights gravitate towards the primary structural elements of these systems and change rarely. Parts which find themselves in flux are spun out into the data, where users can interact with them. Software evolution becomes like a centrifuge stirred by change. The layers that result, over time, can come to a much truer accommodation with the forces that shaped them than any top-down planning could have devised.

Sweeping It Under The Rug

At first glance, a Big Ball of Mud can inspire terror and despair in the hearts of those who would try to tame it. The first step on the road to architectural integrity can be to identify the disordered parts of the system, and isolate them from the rest of it.

Overgrown, tangled, haphazard spaghetti code is hard to comprehend, repair, or extend, and tends to grow even worse if it is not somehow brought under control. If you can’t easily make a mess go away, at least cordon it off. This restricts the disorder to a fixed area, keeps it out of sight, and can set the stage for additional refactoring.

Reconstruction ... alias Total Rewrite

One reason to start again might be that the previous system was written by people who are long gone. Doing a rewrite provides new personnel with a way to reestablish contact between the architecture and the implementation. Sometimes the only way to understand a system it is to write it yourself. Doing a fresh draft is a way to overcome neglect. Issues are revisited. A fresh draft adds vigor. You draw back to leap. The quagmire vanishes. The swamp is drained.

When a system becomes a Big Ball of Mud, its relative incomprehensibility may hasten its demise, by making it difficult for it to adapt. It can persist, since it resists change, but cannot evolve, for the same reason. Instead, its inscrutability, even when it is to its short-term benefit, sows the seeds of its ultimate demise.

The above are highlights from the original article, which is located here ... Big Ball of Mud by Brian Foote and Joseph Yoder [PDF download]

Some further reading on Programmers.StackExchange ...

I've inherited 200K lines of spaghetti code — what now?
How to convince my boss that quality is a good thing to have in code?
How to keep a big and complex software product maintainable over the years?
Techniques to re-factor garbage and maintain sanity?
When is code “legacy”?
What is negative code?

Some related books ...

Working Effectively With Legacy Code ... by Michael Feathers [Amazon]
Refactoring: Improving the Design of Existing Code ... by M. Fowler, K. Beck, et al. [Amazon]
Design Patterns: Elements of Reusable Object-Oriented Software ... by the Gang of Four [Amazon]
Patterns of Enterprise Application Architecture ... by Martin Fowler [Amazon]
Domain Driven Design: Tackling Complexity in the Heart of Software ... by Eric Evans [Amazon]
Head First Design Patterns by E. Freeman, E. Freeman, et al. [Amazon]

Follow @dodgy_coder

Subscribe to posts via RSS

The first right answer is the only answer

2013-01-03T02:16:00.000-08:00

Reading through David B. Stewart's paper entitled "Twenty-Five Most Common Mistakes with Real-Time Software Development" (PDF, 131 KB).

There's an interesting nugget of advice at number 8, "The first right answer is the only answer";

Inexperienced programmers are especially susceptible to assuming that the first right answer they obtain is the only answer. Developing software for embedded systems is often frustrating. It could take days to figure out how to set those registers to get the hardware to do what is wanted. At some point, Eureka! It works. Once it works the programmer removes all the debug code, and puts that code into the module for good. Never shall that code ever change again, because it took so long to debug, nobody wants to break it.

Unfortunately, that first success is often not the best answer for the task at hand. It is definitely an important step, because it is much easier to improve a working system, than to get the system to work in the first place. However, improving the answer once the first answer has been achieved seems to rarely be done, especially for parts of the code that seem to work fine. Indirectly, however, a poor design that stays might have a tremendous effect, like using up too much processor time or memory, or creating an anomaly in the timing of the system if it executes at a high priority.

As a general rule of thumb, always come up with at least two designs for anything. Quite often, the best design is in fact a compromise of other designs. If a developer can only come up with a single good design, then other experts should be consulted with to obtain alternate designs.

As David suggests, when dealing with complex, mission critical or concurrent sections of code, that first success is often not the best solution. Weeks later, you might find that its not performing as well as it should be in production and you have to revisit the section of code again looking for a better solution. But the best time to develop a better solution was back when the job was fresh in your mind, when you developed the original solution.

So he suggests adapting this into a new practice:

As a general rule of thumb, always come up with at least two designs for anything.

There's multiple problems with this advice:

YAGNI, you ain't gonna need it. Develop an extra solution only if and when you need it; don't create extra work where it might not be needed.
Premature optimization is the root of all evil. Knowing when you need to optimize a solution further is almost impossible to do at design time, before any benchmarking or code profiling have been done.
Lastly, if your solution passed the unit test, yet fails further down the line (in production for example), then that suggests there was a problem with your unit test, and not with your solution. If required, you should add some specific concurrency and/or performance testing to your unit test. This will mean you can then optimize your code, while maintaining a TDD approach.

I notice the original article is actually from 1999, which I believe is before test driven development came into prominence. I think this particular piece of advice ("come up with two designs for anything") might have been ok for some projects back then, but would now be considered flawed and certainly not advisable.

Follow @dodgy_coder

Subscribe to posts via RSS

Pricing hacks of online retailers

2012-12-28T00:54:00.000-08:00

1. Free stuff

When retailers offer products and services for "free" there's usually an ulterior motive. For online shops, "free" provides the initial incentive for buyers to use their web site, and may be effective in retaining customer loyalty. One of the most common uses of "free" is free delivery.

Example

UK-based The Book Depository is without doubt using the power of free. The title of their home page is "Free delivery worldwide on all books from The Book Depository". This title shows up in google when searching for "book depository", along with the site's description, which is "The Book Depository offers over 8 million books with free delivery worldwide". The site's byline is "The Book Depository. Free delivery worldwide on all our books". Also in a prominent position on the home page is: "Free worldwide delivery" which links to a page listing all of the countries where they offer free delivery.

2. Per customer limits and the perception of product scarcity

Per customer limits leads buyers to think that a product is scarce, and increases their incentive to buy now. It also entices people to buy more than they originally intended. The legitimate reason for doing this - to avoid products getting sold on the gray market - is hardly ever a real concern. Retailers will also impose a time limit during which the product is available, along with a ceiling on the number of products that can be sold, after which the deal will end.

Example

Living Social Deal - Laser Nail Therapy Clinic - 70% Off Laser Nail Fungus Removal for Feet ($450)
This group buying deal shows a "Limit 4 per customer" in the fine print. Most group buying sites have per customer limits for all their deals. In additon to the per customer limit, there is an absolute limit on how many deals can be sold, as well as the deal only being available for a limited amount of time, in this case, 7 days.

3. The 9 factor - prices ending in 9, 99 or 95

Using prices that end in 9, 99, or 95 is called 'Charm Pricing' or 'Psychological Pricing'. We've been culturally conditioned to associate these prices with discounts. And because we read numbers from left to right, we mentally encode a price like $7.99 as $7, especially when we quickly glance at the price. That's called the "left-digit effect" - it's encoded in our minds before we have finished reading all of the digits.

Example

Pricing of the Amazon Kindle family of products - $499, $199, $119.
As opposed to $500, $200, $120.

4. Easy math

Most sites, when putting a product on sale, will show you what price it was marked down from. It might be "was $20, now $15". You will rarely see something like, "was $20, now $14.22." The reason is that if the difference is easy to calculate, we tend to think it's a better deal. It's called "computation fluency". Another method employed by many sites is to display the amount of the total saving along with the percentage saved.

Example

The Body Shop USA - 50% off Sitewide - After Christmas Sale. Everything is 50% off, making the amount you are saving obvious; you don't need to check any prices or fineprint. Anything you buy on their site will include a significant saving.

5. Sale price font size and color

A common practice in online sales is to display the sale price in a different font size and/or color to the original list price. Typically, the original list price will be shown in strike-though.

Example

Amazon uses red text to denote a sale price, shown below the original list price, which is struck out and shown in light gray. However, products which are not on sale and are being sold at the original list price continue to use this red (sale price) font. One theory says that customers will become accustomed to seeing red text as a saving, and will be more inclined to buy, even when the product is actually not on sale.

6. Dynamic Pricing (also known as Time-Based Pricing)

The airline industry is often cited as a dynamic pricing success story. It employs the technique so well that most of the passengers on any given airplane have paid different ticket prices for the same flight. By responding to market fluctuations or large amounts of data gathered from customers - ranging from where they live to what they buy to how much they have spent on past purchases - dynamic pricing allows companies to adjust the prices of identical goods to correspond to a customer’s willingness to pay.

Example 1

During Thanksgiving week, The New York Times tracked the price of Dance Central 3, a popular Xbox game, as it dropped on Amazon from $49.96 to $24.99 to $15.

Example 2

Coca-Cola tested dynamic pricing in automated vending machines where prices would fluctuate based on the surrounding temperature. Their theory was a soft drink would be worth more when it is hotter outside, and correspondingly, demand for soft drinks would decrease if it were cold outside. It was an unpopular idea and luckily, Coca-Cola abandoned it.

Sidenote

Oren Etzioni, a computer science professor at the University of Washington, became incensed when he found out that the traveller sitting next to him on a flight got a much better price for his ticket than he did. Etzioni started collecting online price data from all US airlines, then he created his own formula that could predict when the airlines would raise or lower their prices. His company was bought by Microsoft. And today, when you search for flights on Microsoft's Bing Travel site, you'll see colour coded arrows (called the "price predictor") letting you know you if the price of that ticket is likely to head up or down.

7. Pay what you want

Pay what you want is a pricing system where buyers pay any desired amount for a given product or service, sometimes including paying nothing (i.e. free). Sometimes a minimum (floor) price may be set, or a suggested price may be indicated to the buyer. The buyer can also pay an amount higher than the standard price.

It has the benefit of reducing buyer's remorse, which is what happens when you decide afterwards that you've paid too much for a product. It also can result in a viral increase in popularity or visibility for a product when used in highly competitive markets.

It is often used for products which use digital delivery, such as software and music. There is usually no additional cost per download to the seller anyway, since the site's bandwidth would usually be paid for on a capped, monthly basis.

Example 1

Freeware software is often distributed under this model. For example, on the home page of Paint.net is the message "Show your appreciation for Paint.NET and support future development by donating!". Typically, a PayPal donate button is used.

Example 2

In October 2007, Radiohead released their seventh album, In Rainbows, through the band's website as a digital download and requested fans just pay whatever amount they thought it was worth.

Example 3

Introduced during May 2010, the Humble Bundle was a set of six downloadable indie games which were distributed using a pay what you want system (with inclusion of a buyer-controllable charitable contribution).

8. Freemium

Freemium is a business model that works by offering a base product or service free of charge (typically digital offerings such as software, games or online software - software as a service). A premium is then charged for advanced features, functionality, or for related products and services.

Example 1

LinkedIn - Basic features of the social network are available for free. For the ability to see the profile of members outside your network, and to access advanced features, you'll need to pay.

Example 2

The New York Times paywall. A limited number of news articles can be read for free per month. To get full access to the site, you'll need to pay.

Example 3

Zynga publishes games for Facebook and the major smartphone/tablet app stores. Typically there are in-game actions (e.g. farming) which result in being rewarded with an in-game currency (e.g. loot/coins/points). The in-game currency can then be used to buy additional game features, or to progress further in the game. The in-game currency can be topped up, via a payment to Zynga in real dollars via an in-app-purchase.

9. No dollar signs

A 2009 Cornell University study found that diners in upscale restaurants spent significantly less when menus contained the word "dollars" or the symbol "$". Restaurants use the technique of omitting dollar signs to get you to focus on the product being sold (the food) rather than the price. They may also mention or profile the chef who is cooking the food. Apart from the websites of restaurants, its difficult to see this being used much in general online retailing.

Example

Gramercy Tavern, New York City
According to Urbanspoon this is the most popular fine dining restaurant in New York City.There's no dollar signs at all on their website, and their a la carte lunch menu (pdf) doesn't feature any dollar signs either, only numbers.

10. "X for $X"

Buyers will often buy more of a product than they originally intended, if it means they will secure a bargain.

Example

Bath and Body Works - 5 for $5 sale
In this case you need to buy 5 products to realise a saving of 33% off the original list price.

Follow @dodgy_coder

Subscribe to posts via RSS

Defensive copy and paste coding

2012-12-16T17:59:00.000-08:00

I've got into the habit of doing this check at the end of any development which has involved some copy & pasting of existing code. It's been useful for me in picking up issues before build/test/commit.

It just involves getting a reference count of all occurrences the new variable (or function) compared with the old variable (or function)...

Select the newly added variable (or function) name
Right-click to bring up the context menu
Select the "Find all references" option

The screenshot shown is from Visual Studio, but this feature is also available in other IDEs and editors, such as Eclipse (via the "References" context menu). Alternatively, if you're not using an IDE, a simple text search across all files in your project will be just as useful, assuming that the search string is unique within your project.

Often, you'd expect the reference count should match with the one you've copy & pasted from (as shown above). If it doesn't match, then by looking at the references which are different, you should be able to easily reason about why, and whether its expected or not. If its unexpected, that could mean there's an error in your code.

So who admits to copy & paste coding - isn't it a sign of bad programming? ... if you can copy & paste big blocks of code, followed by only a few small changes then that's a code smell - you should probably extract the common code into a new function and call it twice. Agreed on that, but there is still many cases where you do have a valid reason to copy existing code. One example is to make fine grained changes to some logic, when it might not be worth the additional complexity of creating a generic function to handle both cases. Or in the case shown above, when I'm copying the declaration of a collection member.

This is a pretty simple and obvious technique, and it might be well known to many already, but I thought I'd mention it anyway. It can also be used outside of the copy & paste case - just mentioning it here because that's the case I've used it more often.

Follow @dodgy_coder

Subscribe to posts via RSS

The slow decline of PCs and the fast rise of Smartphones/Tablets was predicted in 1993

2012-11-04T05:20:00.000-08:00

I've just read some predictions for the future of the PC, written in 1993, by Nathan P. Myhrvold, the former Chief Technology Officer at Microsoft.

His memo is amazingly accurate. Note that his term "IHC" (Information Highway Computer) could be roughly equated with today's smartphone or tablet device, connecting to the Internet via WiFi or a cellular network. In his second last paragraph, Myhrvold predicts the winners will be those who "own the software standards on IHCs" which could be roughly equated with today's app stores, such as those on iOS (Apple), Android (Google, Amazon) and Windows 8 (Microsoft).

The only thing you could say he possibly didn't foresee would be the importance of hardware design in the new smartphone and tablet industry. I'd suggest that Apple achieved such a head start on their competition through a combination of both cutting edge hardware design along with their curated app store model for distributing software. Interestingly, Microsoft has only last month entered the hardware game with their new Surface brand tablets for Windows 8 and Windows RT, and also announced a shift to focus on becoming a "Devices and Services" company.

Note: the term "Cairo" used below is the code name for a Microsoft Research project which lasted from 1991 to 1996. It resulted in some features that were eventually rolled into Windows 95, IIS and SQL Server.

The below is an extract from a memo written by Nathan P. Myhrvold, titled "Road Kill on the Information Highway". September 8, 1993. Full Source.

Personal Computers
I've saved the best for last. Our own industry is also doomed, and will be one of the more significant carcasses by the side of the information highway. The basic tasks that PCs are used for today will continue for a long as it makes sense to predict, so it isn't a question of the category disappearing. The question is one of who will continue to satisfy these needs and how?

As a case in point, consider that the fundamental category needs for mainframes and minicomputers also still exists and will continue to do so for a very long time. Despite this, the companies involved are dying and the entire genre is likely to disappear. The reason is that a new breed of machine - the PC - came along which out flanked them. In the early years PCs were not particularly good at what minis and mainframes did, but they were terrific at a whole new set of problems that the traditional computing infrastructure had basically ignored.

Personal productivity applications drove PCs onto millions of desks and created a very vital industry which grew faster - both in business terms and price/performance - than the mainframe and minicomputer markets. The power conferred by this growth made PCs the tail which wagged the dog; free to ignore the standards which existed for mainframes and minis and move off on their own. Over time the exponential growth in computing has finally (after 17 years) given the PC industry the technical ability to beat minis and mainframes in their own domain. Although the early software platforms for PCs had to be extended to fully realize this potential (DOS to Windows to NT to Cairo), it turned out to be far easier to do this than to make mainframe or minicomputer systems address the new needs and applications. Even within the heart of minicomputer and mainframe's domain - giant transaction processing applications etc., the old standards will not be used.

I believe that the same thing will happen again with PCs playing the role of mainframes and minis, and the computing platforms of the information highway taking over the role of the challenger.

The technical needs of computers on the information highway, or IHCs are quite different than for PCs. The killer applications for IHCs in the early years will include video on demand, games, video telephony and other distributed computing tasks on the highway. It is hard to classify this as either higher tech or lower tech than the software for PCs, because the two are quite different. Most IHCs will certainly need to be cheaper than PCs by an order of magnitude and this will inevitably cause them to be less capable in many ways, but some of their requirements are far more advanced.

Another way to say this is that the rich environment of software for PCs is largely irrelevant for IHCs. Windows, NT, System 7 and Cairo do not solve the really important technical problems required for IHC applications, and it is equally likely that the early generations of IHC software won't be great platforms for PC style apps. This isn't surprising because they are driven by an orthogonal set of requirements.

The IHC world will almost certainly grow faster than PCs, both in business terms and in price/performance. The PC industry is already reaching saturation from a business perspective. Technically speaking, the industry is mired in hardware standards (Intel and Motorola CISC processors) with growth rates that are flattening out relative to the state of the art - just as the 360/3090 and VAX architectures did. The Macintosh and Windows computing environments may be able to survive the painful transition to new RISC architectures, but they will lose time and momentum in doing so.

PCs will remain paramount within their domain for many years (we'll still have a computer on every desk) but IHCs will start to penetrate a larger and larger customer base on the strength of its new and unique applications. The power of having the worlds information - and people - on line at any time is too compelling to resist. For a long time people will still have a traditional PC to handle traditional PC tasks - in precisely the same way that they have kept their mainframes and minis for the last 17 years. One day however people will realize that their little IHCs are more powerful and cheaper than PCs - just as we have finally done with mainframes. There will be a challenge for the IHC software folks to write the new systems and applications software necessary to obviate PCs, just as we had to work pretty hard to come up with NT, but this battle will clearly go to the companies who own the software standards on IHCs. The PC world won't have any more say about how this is done than the companies who created MVS or VMS did about our world. Of course, some of the VMS people were involved, but as discussed above it is very hard for organizations to make the transition.

This may sound like a rather dire prediction, but I think that for the most part it is inevitable. The challenge for Microsoft is to be sufficiently involved with the software for the IHC world that we can be a strong player in that market. If we do this then we will be able to exploit a certain degree of synergy between IHCs and PCs - there are some natural areas where there is benefit in having the two in sync. The point made above is that those benefits are not sufficiently strong that they alone will give us a position in the new world. We'll live or die on the strength of the technology and role that we carve out for ourselves in the brave new world of the information highway.

Many thanks to Reddit user erpettie who originally submitted a link to this memo on /r/technology,which is how I came across it.

Follow @dodgy_coder

Subscribe to posts via RSS

Q&A With Nine Great Programmers

2012-09-08T06:50:00.000-07:00

This Q&A article is originally from 2006, published on the blog of Polish developer Jaroslaw "sztywny" Rzeszótko (AKA "Stiff"). That original blog post is no longer online. Gratefully republished with his permission.

On a hot, boring afternoon I got an idea. With the help of publicly accessible e-mail addresses, I asked 10 questions to a bunch of programmers that I consider very interesting people and I respect them for various things they created. Coming out with the questions was a 5 minute job for me - these are things I would ask about if I could speak with them personally for, let’s say, 10 minutes, and I didn’t have time for thinking too much. The last two question don’t have anything to do with programming, this is simply something I like to know about everyone I talk to, lets say that’s my hobby. Not everyone wanted to answer them, and that’s fine. It was the first "interview" I ever made, so I also made some mistakes, which went out as people started answering. But despite this, I learnt a lot of interesting stuff, so it was definitely a valuable experience.

Not everyone responded to my e-mail, not everyone agreed to answer the questions, maybe I will also get some answers after I published this, I didn’t have the patience to wait longer, so new things may appear here over time (Update: Bjarne Stroustrup was added on 03.08.2006).
-- Jaroslaw

Starring

Linus Torvalds - The Linux kernel author.

Dave Thomas - Author of the "Pragmatic Programmer", "Programming Ruby" and other great books about programming. One can read his mainly programming-related thoughts here.

David Heinemeier Hansson - Author of the Rails Framework - the new hot web development framework. He has a blog here.

Steve Yegge - Probably one of the least known guys here, but he also sent most interesting answers. He has a popular blog about programming. He is also the author of a game called "Wyvern".

Peter Norvig - Research Director at Google, a well known Lisper, author of famous (in some circles at least) books about AI. See his homepage here.

Guido Van Rossum - The Python language creator.

Bjarne Stroustrup - C++ language creator, has a homepage here.

James Gosling - The Java language creator.

Tim Bray - One of the XML and Atom specifications author and a blogger too.

Q 1: How did you learn programming? Were any schools of any use? Or maybe you didn’t even bother with ending any schools :) ?

Steve Yegge
I taught myself to program on an HP calculator using their RPN stack language when I was 17 years old. I’d tried to learn programming a few times before that but never really "got" it. The HP 28c and 48g scientific calculators were pretty powerful and had great docs. I wrote a 3D wireframe viewer for the 48g — I got a book on 3D graphics and painstakingly translated an example program in Pascal into the RPN stack language. It was pretty sweet when I got it running. After that I bought a PC and Turbo Pascal, and started studying programming in earnest. I was a decently good programmer by the time I went into the CS program in college.

I went to the University of Washington and got an undergrad degree in CS. It was definitely worthwhile, and I recommend that all programmers should try to get a CS degree if possible.

Linus Torvalds
I didn’t learn programming in school, but mostly on my own reading books and just doing it (initially on a Commodore VIC-20, later on a Sinclair QL).

That said, I think especially University was very useful. Rather than go to an engineering school, I went to Helsinki University, which is pretty theoretical, so there the teaching concentrated not so much on programming (which was just a small part, and which I ended up doing more of "on the side" anyway), but most of the courses tended to be on fundamental concepts and things like complexity analysis. Which can seem boring and even a waste of effort at times, but I think it was useful, and I mostly enjoyed it. And I think I’m probably a better programmer for it.

David Heinemeier Hansson
I learned programming by starting to put together my first web page in HTML. Then I wanted to make some dynamic pieces and picked up first ASP then PHP. After I already knew how to program, I then started on a joint computer science and business administration degree.

Peter Norvig
I took courses in high school and college, but always felt I learned more on my own.

Dave Thomas
During my secondary schooling I took a class in a local technical college on computers. It got me totally hooked: I fell in love with programming, and looked around for colleges offering courses in software. Eventually I went to Imperial College, part of London University. It was only the second year they’d offered a course in software, and it was absolutely marvelous: the staff and students worked together to make the materials better, and everyone learned a lot. The undergraduate course there gave me an incredibly strong background in software development. I stayed on to start a PhD, but got lured away by a startup.

But the overall question is "how did you learn programming?" The real answer to that is "I’m still learning programming." I think any good developer continues to learn throughout their careers. It isn’t just a question of picking up new languages and libraries: good developers also refine their techniques and practices over the years.

Guido Van Rossum
I went to university where they had a big mainframe and there were various computer courses. This was very important for me.

James Gosling
Initially, I was self-taught. I got my first programming job before I went to college. But I’m glad I did. I had a lot of fun. I kept going until I had a PhD.

Bjarne Stroustrup
In university (Aarhus and later Cambridge). The universities taught me much that was useful, incl. most of the bases for my future work. In addition, I learned a fair bit from programming for money - where understanding of real-world problems, correctness, maintainability, on-time delivery, etc. were more pressing than in an educational setting.

Tim Bray
I thought I was going to be a math teacher. The math program at University required a few computer science courses.

Q 2: What do you think is the most important skill every programmer should posses?

Steve Yegge
Written and verbal communication skills. You’ll never make it very far as a programmer in any field unless you can get your ideas across to people effectively. Programmers should read voraciously, practice writing, take writing courses, and even practice at public speaking.

Linus Torvalds
It’s a thing I call "taste".

I tend to judge the people I work with not by how proficient they are: some people can churn out a lot of code, but more by how they react to other peoples code, and then obviously by what their own code looks like, and what approaches they chose. That tells me whether they have "good taste" or not, and the thing is, a person without "good taste" often is not very good at judging other peoples code, but his own code often ends up not being wonderfully good.

But hey, it’s not the only thing. One thing that is very useful, especially in an open source project, is simply the ability to communicate well what you want to do, and how you are going to do it. The ability to explain to others why you do something a certain way is very important, and not everybody has that ability.

That said, in the end there are also the people who just churn out good code. They may not be good at explaining it, and they may not even have great taste, but the code works well. Sometimes you need another person (one that does have that hard-to-define "taste") to maybe massage the code into a form where it’s useful in the bigger picture, but just the ability to write clear code for difficult problems is obviously a fairly fundamental part of any programmer.

David Heinemeier Hansson
A strong sense of value. The ability to ask yourself the question: Is it worth doing what I’m doing right now? So many programmers seem to waste oceans of time on stuff that just doesn’t matter. And not enough on the stuff that does.

Peter Norvig
I don’t think there’s one, but let’s say concentration.

Dave Thomas
Passion.

Guido Van Rossum
Your questions are rather general and hard to answer. :-) I guess being able to cook an egg for breakfast is invaluable.

James Gosling
To be self motivated. To be really good, you have to be in love with what you do.

Bjarne Stroustrup
The ability to think clearly: A programmer has to understand problems and express solutions.

Tim Bray
Ability to prefer evidence to intuition.

Q 3: Do you think mathematics and/or physics are an important skill for a programmer? Why?

Steve Yegge
There is a large branch of mathematics that’s very important for programmers called "discrete math" or "concrete math". It includes disciplines such as probability, combinatorics, graph theory, induction proofs, and other useful tools. I would encourage all programmers to study discrete mathematics to whatever extent they can. Even a little is better than none at all.

As for more traditional math, well, I don’t use it as often, but it comes in very handy when I need it. For instance, I’ve only used calculus once in the past year as part of my job. I had to estimate loads for the peak traffic hour of the day for a service whose load "follows the sun" in an approximate sine curve. The simplest way to make the estimate was to integrate over 1/24th of the curve at a specific time. If I hadn’t known calculus, I would not have known how to make a reasonably accurate estimate.

When I was writing my game, Wyvern, having a solid working knowledge of basic planar geometry was incredibly helpful. And it’s quite common to use algebra and linear algebra on a regular basis. But I rarely use trigonometry or differential equations on the job, and not much calculus either.

I’d say my basic math foundation has made me maybe 5% to 10% better as a programmer. If I knew a lot more math, I’d undoubtedly be a much better programmer than I am today, so I study and practice math several hours a week.

I love physics and I have an ongoing, lifelong quest to try to understand the underpinnings of quantum mechanics. But I’ve never personally found any physics very useful towards my job as a programmer. That would, of course, be diffferent if I were doing something in a physics domain, such as 3D game programming, or certain types of simulation.

Linus Torvalds
I personally think a fairly strong math background is a good thing. I’m not as sure about the physics side, but I’m convinced that understanding math and having a good background in it helps you to be a better programmer. If only because the mental models are similar - you can build up any kind of set of rules you want, but it should be self-consistent.

David Heinemeier Hansson
Not at all. At least not for the kind of business programming needed for web applications. I consider it much more important that someone is a good writer.

Peter Norvig
Yes. Many ideas are inherently mathematical: induction, recursion, logic, etc.

Dave Thomas
Maybe. But, to be honest, I haven’t seen much of a correlation either way between these types of discipline and good software developers.

However, I have seen a strong correlation between people who have some music in their background and programming skills. I have no idea why, but I suspect that some of the areas of the brain that make someone musical also make them good at software development.

Guido Van Rossum
Math, yes (for some parts; I don’t care for differential equations, but algebra and logic are important). Physics, I don’t think so except it’s always useful to be interested in many different things.

James Gosling
Yes! They teach you logic & deduction... To have an analytical eye. And there’s no replacement for mathematics when it comes to analyzing algorithms.

Bjarne Stroustrup
Depends on the programmer and the programming tasks. Some forms of math are frequently useful; physics less often so, but then learning physics is one of the best ways of learning practical math.

Tim Bray
In my case, I’ve almost never used my university-level math to support my programming.

Q 4: What do you think will be the next big thing in computer programming? X-oriented programming, y language, quantum computers, what?

Steve Yegge
I think web application programming is gradually going to become the most important client-side programming out there. I think it will mostly obsolete all other client-side toolkits: GTK, Java Swing/SWT, Qt, and of course all the platform-specific ones like Cocoa and Win32/MFC/etc.

It’s not going to happen overnight. It’s very slowly been going that direction for ten years, and it could well be another ten years before web apps "win". The tools, languages, APIs, protocols, and browser technology will all have to improve far beyond what you can accomplish with them today. But each year they get a little closer, and I’ve finally decided to switch all my own app development over to browser-based programming from now on.

Microsoft and Apple definitely don’t want this to happen, so a necessary first step will be for an open-source browser such as Firefox to achieve a dominant market position, which will in turn require some sort of Firefox-only killer app. (A killer app would be something like iTunes, something that everyone in the world wants to use, badly enough to download Firefox for it.)

Linus Torvalds
I don’t think we’ll see a "big jump". We’ve seen a lot of tools to help make all the everyday drudgery easier - with high-level languages and perhaps the integration of simple databases into the language being the main ones. But most of the buzz-words have been of pretty limited use.

For example, I personally believe that "Visual Basic" did more for programming than "Object-Oriented Languages" did. Yet people laugh at VB and say it’s a bad language, and they’ve been talking about OO languages for decades.

And no, Visual Basic wasn’t a great language, but I think the easy DB interfaces in VB were fundmantally more important than object orientation is, for example.

So I think there will be a lot of incremental improvements, and the hardware improvements will make programming easier, but I don’t expect any huge productivity help or revolutions in how people do things.

At least not until you start approaching real AI, and I don’t think real AI is going to be anything you will ever "program".

David Heinemeier Hansson
I try not to predict the future. I’m not a big believer in fortune telling. The best way to predict the future is to implement it.

Peter Norvig
Large-scale distributed processing.

Dave Thomas
The next big thing in computer programming will be eclipsed by the next–next big thing in programming, and so on, and so on. I’m kinda tired of the endless search for the big things, because while doing it people tend to forget about the real issues: getting the fundamentals right. We need to get a whole lot better at talking with our customers, focussing on delivering value, and taking pride in what we do. A developer who can do these things can deliver great software with any tool set, and won’t need to worry about tracking the fads and fashions.

Guido Van Rossum
Sorry, I’m not much of a crystal ball person. I predicted CGI about 5 years after it had been invented. :-)

James Gosling
The two issues I’m most concerned about now are coping with parallelism and complexity.

Bjarne Stroustrup
I don’t know, and I don’t like to guess.

Tim Bray
No idea.

Q 5: If you had three months to learn one relatively new technology, which one would you choose?

Steve Yegge
I do happen to have 3 months (part-time), and I’m spending it learning Dojo (http://dojotoolkit.org) and advanced AJAX and DHTML. I’m learning it by writing a fairly ambitious web application. Dojo’s really cool, and I’m sure it will only improve with time.

Linus Torvalds
Hmm. I’d really love to do FPGA’s, but I’ve always been too busy to really sit down and start learning. I love the notion of playing with hardware: it’s obviously one of the reasons I ended up doing operating systems, since that (along with compilers) is about as close as you can get to playing with the hardware, without actually designing or building it yourself.

David Heinemeier Hansson
Cocoa programming for the Mac.

Peter Norvig
I’d like to know Javascript better. Also flash.

Dave Thomas
If "new" means "new to Dave Thomas" then I think I’d take intensive piano lessons.

If "new" means technology stuff, then I guess I’d choose technologies related to accessibility for people with disabilities.

Guido Van Rossum
Snowboarding.

James Gosling
For fun, I’d catch up on the latest in 3D rendering. I’d probably write a photon-map renderer.

Bjarne Stroustrup
There are very few things of any importance that you can learn in three month. I think you must be thinking of training in a well established field.

Tim Bray
Security, encryption, digital signatures, identity, etc. It’s a big problem for me that I’ve never learned this stuff.

Q 6: What do you think makes some programmers 10 or 100 times more productive than others?

Steve Yegge
I think if you pause to consider why not all atheletes are equally good, you’ll have your answer(s). Thomas Edison has a relevant quote about genius that might also provide you some clues.

Linus Torvalds
I really have no idea. I think some people are just better able to concentrate on the things that matter, and I think a lot of it is just doing it. Most of the really good programmers I know started doing it fairly young.

David Heinemeier Hansson
The ability to restate hard problems as easy ones.

Peter Norvig
The ability to fit the whole problem into their heads at one time.

Dave Thomas
They care about what they do.

Guido Van Rossum
Genetically different brain structure.

James Gosling
They think about what they do. They don’t rush in and slap things together. They have a holistic picture of what is to be built.

Bjarne Stroustrup
First a general lack of professionalism and adequate training. that sets the base level too low. Secondly, some people have a combination of "smarts" (ability to think clearly and get to the heart of things), experience, and knowledge of tools. Programming leaves more scope for that because it is a combination of theory and practice - neither of which is much use without domain knowledge.

Tim Bray
The surprising variability of the human mind.

Q 7: What are your favourite tools (operating system, programming/scripting language, text editor, version control system, shell, database engine, other tools you can’t live without) and why do you like them more than others?

Steve Yegge
OS: Unix! I use linux, cygwin, and darwin all about equally often now. You just can’t beat it for productivity tools. Every programmer should learn how to use every tool in /bin and /usr/bin.

Scripting language: Ruby. I’m proficient with just about every major scripting language out there: Perl, Python, Tcl, Lua, Awk, Bash, and others I’m forgetting. But I’m really lazy, and Ruby’s by far the easiest, so it’s a match made in heaven.

Programming language: I don’t have a favorite; I think they all suck. I tend to prefer Java because it’s a strong, portable platform with good tools and good libraries. But the Java language will evolve or die; it’s not good enough as-is to hold the lead indefinitely.

Text editor: Emacs, because it’s the best thing out there today.

Version control: SVN. Perforce is better, but it’s very expensive.

Shell: Bash, because I’m too lazy to learn a better one.

Database engine: MySQL, of course. Nothing else comes close.

Others: I find the GIMP invaluable, and also maddeningly unintuitive. I’ve been using it for years and can still barely do anything with it. But I couldn’t live without it, ironically enough.

Firefox is becoming an increasingly critical part of my tools lineup. I feel suffocated when I’m forced to use IE or Safari.

Note that all these tools (Unix, Emacs, Firefox, GIMP, MySQL, Bash, SVN, Perforce) have something in common: they’re extensible; i.e., they all have programming APIs. Great programmers learn how to program their tools, not just use them.

Linus Torvalds
I actually don’t end up having that many tools I work with, and for the many of them I have spent some time of my own to just make them work for me. The OS part is clearly the biggest one, but I’ve obviously also written my own version control system (git), and the text editor I use (micro-emacs) I’ve ended up customizing and extending upon too.

Other than those three parts, the only thing I care deeply about is my email reader. I use "pine" - not because it’s necessarily the greatest email reader ever, but because I’m used to it, and it does what I need it to do with a minimum of fuss.

David Heinemeier Hansson
OS X, TextMate, Ruby, Subversion, MySQL. That’s the combo currently keeping me happy. I like tools that exhibit good taste and a focus on the stuff that matters.

Peter Norvig
I dislike all three major OS - Windows, Mac, Linux. I like Python and Lisp. Emacs.

Dave Thomas
I switched to Macs a couple of years ago after being a Linux person for more than 10 years. The tools are not necessarily better, but they don’t have to be sharpened or maintained as often, which lets be concentrate on just using them.

I'm not a great believer in single tools: I tend to switch around quite frequently just so I can get experience with as many tools as possible. Right now I’m using OSX, Emacs, TextMate, Rails, Ruby, SVN, CVS, Rake, make, xsltproc, TeX, MySQL, Postgres, and a whole lot of small productivity aids. Who knows what I’ll be using next year.

Guido Van Rossum
Unix/Linux, Python, vi+emacs, Firefox.

James Gosling
These days I live in NetBeans. It does everything I want, very cleanly simply and efficiently. It’s the nicest environment I’ve ever lived in.

Bjarne Stroustrup
Unix, sam (a very simple text editor), and a good C++ compiler, of course.

Tim Bray
I like Unix-like operating systems, dynamic languages like Python and Ruby and statically-typed languages like Java (in particular the Java APIs), Emacs, whatever, bash, whatever, NetBeans.

Q 8: What is your favourite book related to computer programming?

Steve Yegge
Man, that’s a tough one. Maybe "Gödel, Escher, Bach: an Eternal Golden Braid" (Hofstadter)? Although it’s not strictly about programming. If you specifically mean "favorite book about programming", then maybe SICP (mitpress.mit.edu).

Linus Torvalds
Heh. When I read these days, I tend to either read fiction, or non-computer-related stuff (oldie but goodie: "The Selfish Gene" by Richard Dawkins).

When it comes to programming, the only real programming book that comes to mind is actually the classic Kernighan & Ritchie "The C Programming Language" book, because it’s such an incredibly useful book while being so very readable and short. Considering that you can basically learn one of the most important programming languages of our times from it, the fact that it’s thin and readable is just a wonder.

That said, many other books I enjoyed a lot were not about programming per se, but about computer architecture and hardware. There’s obviously Patterson & Hennessy’s computer architecture book, but for me personally perhaps even more Crawford & Gelsinger’s "Programming the 80386?, which was what I used when I started with Linux.

For similar reasons, I have a soft spot for Andrew Tanenbaum’s "Operating Systems: Design and Implementation".

David Heinemeier Hansson
I like Extreme Programming Explained for its rejection of common thinking about programming practices and Patterns of Enterprise Application Architecture for striking the right balance of abstract and concrete.

Peter Norvig
Structure and Interpretation of Computer Programs

Dave Thomas
It depends on what you mean by "favorite." Probably the best written book I’ve read in the area is IBM’s "IBM/360 Principles of Operation."

Guido Van Rossum
Neil Stephenson’s Quicksilver.

James Gosling
Programming Pearls by Jon Bentley.

Bjarne Stroustrup
K&R.

Tim Bray
Bentley’s Programming Pearls

Q 9: What is Your favourite book NOT related to computer programming?

Steve Yegge
Just one book? You’re asking for the impossible. There are too many great books out there to choose just one.

My favorite books that I’ve read this month are "Stardust" (Neil Gaiman) and "The Mind’s I" (Hofstadter/Dennet).

My favorite writers are Kurt Vonnegut, Jr. and Jack Vance.

Linus Torvalds
Well, I already mentioned the Selfish Gene by Dawkins. On the fictional side, there’s just a lot of books I’ve read and enjoyed, but few I’d say were my "favourite" one. I tend to not often re-read the books, and the selection would change over time. It’s mostly science fiction and fantasy, eg "Stranger in a Strange Land" by Heinlein was my favourite one as a teenager, but it’s a bit less clear-cut for me these days..

David Heinemeier Hansson
1984, George Orwell.

Guido Van Rossum
Neil Stephenson’s Quicksilver.

James Gosling
Guns, Germs & Steel by Jared Diamond

Bjarne Stroustrup
It changes over time. Currently O’Brian’s Aubrey/Maturin series.

Tim Bray
One Day in the Life of Ivan Denisovich

Q 10: What are your favourite music bands/performers/composers?

Steve Yegge
Favourite genres: classical, anime soundtracks, video-game music
Favourite composers: Rachmaninoff, Chopin, Bach
Favourite performers: David Russell (classical guitar), Sviatoslav Richter (piano)
Favourite anime OSTs: Last Exile, Haibane Renmei

Linus Torvalds
I'm actually not very much into music, but when I listen to it, I tend to listen to various classic-rockish things, ranging from Pink Floyd to the Beatles to Queen and The Who.

David Heinemeier Hansson
I like a lot of genres. Beth Orton, Aimee Mann, Jewel, Lauryn Hill. Actually, all those examples would fit under Girls with Guitars ;).

Guido Van Rossum
Philip Glass.

James Gosling
I tend to like folk musicians: Christine Lavin, Woody Guthrie, Pete Seeger...

Bjarne Stroustrup
Band: The Dixie Chicks. Composer: Beethoven.

Tim Bray
Read my blog.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Postscript

How I found this: I was reading this post by Jeff Atwood, titled "Linus Torvalds, Visual Basic Fan" where the source article he linked to was: "STIFF ASKS, GREAT PROGRAMMERS ANSWER". But that link was dead, and doing a search didn't find any trace of it anywhere on the net. So I resorted to an archive.org search and sure enough, up popped this incredible blog post by Jaroslaw Rzeszótko.

Due to it no longer being available on the net, I thought it should reposted again so others can enjoy reading it. I requested permission from the original author to do this, and he happily obliged - thanks again Jaroslaw!

Follow @dodgy_coder

Subscribe to posts via RSS

Read a Chinese translation of this blog post << Thanks to the guys at CoolShell.cn for this!

Read a Turkish translation of this blog post << Thanks to Seval for this!

Startups for Dummies

2012-08-24T21:45:00.000-07:00

The problem with Steve Blank’s How to Build a Web Startup (Lean Launchpad Edition) article ...

There’s little to no focus on the actual product being built.

Only 1 out of 50 points listed is actually about building a product: Its listed under step 7, which is: “Add the backend code to make the site work”.

The rest is mostly cruft and admin type work that an outsourced worker could do. No doubt its important, but 98% compared to 2%?

The article comes across as a guide to building an over-hyped marketing vehicle, solely to receive funding, rather than actually building something useful that people want to use. There’s no mention of innovation or of making people’s lives easier. This is Startups For Dummies.

Full respect to Steve Blank (a 34 year veteran of Silicon Valley) and his views, but in my view, this particular article has a glaring weakness.

Some further reading on the topic:

This Starting a Startup article by Paul Graham is simple but contains three great points he says you should focus on:

People/a good team
A product that customers want
Low overheads

This 12 Ways To Make Your Web Startup Investable article by StartupSmart, recommends:

People/a good team
A clear market gap (i.e. a product that customers want)
Timing is everything

Incidentally, Steve Blank has just announced he is giving a free online course over at Udacity - Entrepreneurship: The Lean Lanchpad (EP245). It starts on 14 September, 2012.

Follow @dodgy_coder

Subscribe to posts via RSS

TCP/IP Sockets in .NET 4

2012-08-23T05:37:00.000-07:00

TCP is one of the core protocols of the Internet and is a bi-directional, client/server protocol. In .NET you can use the TcpListener Class for the server component, which provides simple methods that listen for and accept incoming connection requests from clients. You can use the TcpClient Class for the client component, which provides client connections to TCP server components; it provides simple methods for connecting, sending, and receiving stream-based data over the network.

Here's a standalone C# 4.0 app that demonstrates how to setup a TCP/IP sockets based client and server in .NET. It uses a command line option to determine whether to run the client or server. There's no external files or libraries needed, just the standard ones you get with a C# 4.0 console project. I've tested it with Visual Studio 2010 in Windows, and it will also work unchanged with Mono.

To run the server, from the command prompt...
SocketsTest -server

To run the client, from another command prompt ...
SocketsTest -client

Below is the source code ...


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Threading.Tasks;

/// 
/// Standalone TCP/IP sockets example using C#.NET 4.0
/// 

/// 
/// Compile as a C#.NET console mode application
/// 
/// Command line usage:
///   SocketsTest -client   => run the client
///   SocketsTest -server   => run the server
/// 
class SocketsTest
{
    static TcpListener listener;

    // Sample high score table data
    static Dictionary<string, int> highScoreTable = new Dictionary<string, int>() { 
            { "john", 1001 }, 
            { "ann", 1350 }, 
            { "bob", 1200 }, 
            { "roxy", 1199 } 
    };

    static int Port = 4321;
    static IPAddress IP_ADDRESS = new IPAddress(new byte[] { 127, 0, 0, 1 });
    static string HOSTNAME = "127.0.0.1";
    static int MAX_CLIENTS = 5;

    public static void Main(string[] args)
    {
        if (args.Contains("-server"))
        {
            ServerMain();
        }
        else if (args.Contains("-client"))
        {
            ClientMain();
        }
        else
        {
            Console.WriteLine("Usage: SocketsTest -client   => run the client");
            Console.WriteLine("       SocketsTest -server   => run the server");
        }
    }

    /// 
    /// Server receives player name requests from the client and responds with the score.
    /// 

    private static void ServerMain()
    {
        listener = new TcpListener(IP_ADDRESS, Port);
        listener.Start();
        Console.WriteLine("Server running, listening to port " + Port + " at " + IP_ADDRESS);
        Console.WriteLine("Hit Ctrl-C to exit");
        var tasks = new List();
        for (int i = 0; i < MAX_CLIENTS; i++)
        {
            Task task = new Task(Service, TaskCreationOptions.LongRunning);
            task.Start();
            tasks.Add(task);
        }
        Task.WaitAll(tasks.ToArray());
        listener.Stop();
    }

    private static void Service()
    {
        while (true)
        {
            Socket socket = listener.AcceptSocket();

            Console.WriteLine("Connected: {0}", socket.RemoteEndPoint);
            try
            {
                // Open the stream
                Stream stream = new NetworkStream(socket);
                StreamReader sr = new StreamReader(stream);
                StreamWriter sw = new StreamWriter(stream);
                sw.AutoFlush = true;

                sw.WriteLine("{0} stats available", highScoreTable.Count);
                while (true)
                {
                    // Read name from client
                    string name = sr.ReadLine();
                    if (name == "" || name == null) break;

                    // Write score to client
                    if (highScoreTable.ContainsKey(name))
                        sw.WriteLine(highScoreTable[name]);
                    else
                        sw.WriteLine("Player '" + name + "' was not found.");

                }
                stream.Close();
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
            }

            Console.WriteLine("Disconnected: {0}", socket.RemoteEndPoint);
            socket.Close();
        }
    }

    /// 
    /// Client requests a player name's score from the server.
    /// 

    private static void ClientMain()
    {
        TcpClient client = new TcpClient(HOSTNAME, Port);
        try
        {
            // Open the stream
            Stream stream = client.GetStream();
            StreamReader sr = new StreamReader(stream);
            StreamWriter sw = new StreamWriter(stream);
            sw.AutoFlush = true;

            // Read and output the first line from the service, which
            // contains the number of players listed in the table.
            Console.WriteLine(sr.ReadLine());

            while (true)
            {
                // Input player name
                Console.Write("Enter player name: ");
                string name = Console.ReadLine();

                // Write name to server
                sw.WriteLine(name);
                if (name == "") break;

                // Read score from server
                Console.WriteLine(sr.ReadLine());
            }
            stream.Close();
        }
        finally
        {
            // Close the connection
            client.Close();
        }
    }
}

Follow @dodgy_coder

Subscribe to posts via RSS

EMPORI: The original Amazon Locker from 12 years ago

2012-08-11T23:29:00.000-07:00

Amazon Locker is a new delivery option offered by Amazon which provides secure, self-service pick-up stations located in a select number of cities in the US and UK. Once your package is delivered to the Amazon Locker, you receive an e-mail informing you that your package is available for pick-up.

This Amazon service came about 1 year after a Canadian startup called BufferBox started offering a similar secure dropbox service. BufferBox has lockers installed in two locations in Waterloo, one at the university and another a nearby corporate office. The company is nearing a launch in Toronto and plans to roll out lockers across Canadian and U.S. cities.

But back in 2000, another Canadian company launched a similar service. On July 24, 2000, Canadian property management company Oxford Properties Group launched a startup called EMPORI.COM. Customers could order products from affiliated online retailers via Empori.com, which were then delivered on the same or next day to a secure depot containing lockers. The original depot was located in an office building in Toronto (Royal Bank Plaza) and over the course of a year this expanded to three more sites around Toronto. Oxford Properties Group saw this as a way for their commercial real estate customers to get additional value out of their prime inner city locations. The idea was that city workers would order products online and then, after receiving an email notification with the locker location and locker combination, would pick their order up at the end of the day, on their way home from work. A host of a products from 31 vendors were offered, from books to running shoes to groceries and even liquor.

The upmarket secure lockers offered by Empori

From the launch press release: "Empori.com is the one-of-a-kind retail solution to those barriers that discourage people from purchasing online. Our delivery system addresses the online shopping fulfillment problem."

Less than 1 year later, on July 17, 2001, Empori.com closed after a $5 million loss. The blame was initially placed on the general business environment which included the bursting of the dot com bubble: "With recent changes in the dot-com environment, [Empori] was unable to find another strategic and financial partner to support further development of the business-to-consumer business,"

As the dust settled it looked like the single biggest factor in Empori’s failure was its inability to secure a critical mass of online retailers. They tried to do too much - secure depots for pick-up was only one piece of their ambitious plan - they were also reselling goods via their website, which was the sole point of contact to access their secure depots. In effect they were trying to launch a new online retail site, coming off a base of zero customers, while in the general dot com gloom of 2000/01. Maybe what they should have done is to focus solely on their unique selling point - the secure depot - and offered it as a service, available to any online retailer, much as BufferBox is doing today.

The business burned $5 million in one year before closing

As of today, the Empori.com domain name is offered for outright sale for €3540 ($4351) by a European domain reseller. "Empori" is Italian for "Stores".

Follow @dodgy_coder

Subscribe to posts via RSS

Old school developers - achieving a lot with little

2012-07-21T03:11:00.000-07:00

Ken Thompson

I was reading through bits of the excellent Coders at Work book by Peter Seibel and found it interesting that the (what I'd call respectfully) old school developers, such as Ken Thompson, Joe Armstrong and Jamie Zawinski use hardly any modern tools and techniques while developing software.

Ken Thompson was the designer of the B programming language (the precursor to C) and also the Go programming language (with Rob Pike). He also worked alongside Dennis Ritchie to develop the C programming language and the original Unix operating system while working at Bell Labs. After designing the UTF-8 method of character encoding on a placemat in a New Jersey diner, he worked overnight to implement full UTF-8 support in Bell Lab's Plan 9 operating system. Regarding his programming style, he debugs only via printf statements, hardly ever uses unit tests, starts his projects by designing the data structures and then works bottom up, with throwaway test stubs.

Joe Armstrong

Joe Armstrong developed the Erlang programming language and the open source Open Telecom Platform (OTP) framework for Ericsson. When designing software he prefers to rigorously document as much as possible up front before starting to write code, especially for difficult projects such as those involving real-time networking protocols. He uses prototypes to solve the hard problems first, and for debugging, just uses print statements. He is a critic of Object Oriented Programming, and favours functional programming languages like Haskell. He never uses an IDE, preferring just Emacs and the command line (no mouse required).

Jamie Zawinski worked in MIT's AI lab with Lisp, then moved onto leading the development of Lucid Emacs, also in Lisp, which later became known as XEmacs. After this he joined Netscape and worked on the front-end for the Unix version of Netscape Navigator, and later led the development of Netscape Mail (both projects were in C). He too prefers to just use print statements to debug his code. His process is either to write code top down or bottom up, let the code evolve naturally, then refactor it when necessary. During development he hardly ever uses unit tests, believing it slows things down - he thinks there's a lot to be said for getting the code right first time. In his view, its a matter of priorities, "do you want this to be good software or do you want it to be done next week - pick one because you can't have both".

Jamie Zawinski

So what tools and techniques do you actually need to be a great programmer? Are these modern techniques of TDD, BDD, Scrum, Agile, Design Patterns, XP etc. just window dressing or do they serve a higher purpose? Is it just a fact that a gifted programmer will always find a way to be a great programmer, no matter what tools he or she works with?

My own view is that some level of TDD and a reliance upon a decent set of unit tests is invaluable. Maybe then these modern techniques should just be seen as advances to make our job a little easier and more productive, especially for those who are not a genius-level developer, like these guys.

Follow @dodgy_coder

Subscribe to posts via RSS

Being a great tester

2012-06-27T04:49:00.000-07:00

I just wanted to post a great little nugget of information from the end of Unit 1 of Software Testing: How to Make Software Fail (CS258) on Udacity, which is being taught by John Regehr and Sean Bennett.

The developer's attitude is "I want the code to succeed".
The tester's attitude is "I want the code to fail".
Although they seem contradictory, understand that you are both aiming to create more robust, higher quality software.
The developer aims to write quality code first time.
The tester aims to identify bugs which can then be fixed, thereby improving the robustness of the code.

Great testers ...

Test creatively - use their knowledge of the software under test to be creative in their test methods and test structure.
Don't ignore small glitches that are easy to overlook or ignore, but may hint at larger problems, which when followed up can often lead to major issues.

Watch the snippet on YouTube: Being a great tester (2 min, 33 sec)

Follow @dodgy_coder

Subscribe to posts via RSS

Applying hardware testing concepts to software

2012-06-03T01:52:00.000-07:00

I've recently been testing a desktop application I built with WPF and .NET 4 which collects data from the serial port and draws various charts on screen in real time. The volume of data isn't immense, one data frame of approx 200 bytes arriving every 200 milliseconds, but there is additional processing happening in the background. One of the steps being performed is regression analysis on various data points to construct curves of best fit.

In production this process will be run for likely not more than 30 minutes, but during testing I wanted to test the limits of the application by running the process for 2 hours, to be sure there was no memory leaks or performance issues. I was surprised when after just 1 hour the UI just became totally unresponsive, dramatically quickly.

I did find the issue and made the fix, but this got me thinking about where else this sort of issue could occur in software - because all I'd done was apply a simple form of software performance testing, known as endurance testing, which just involved running the software for significantly longer than normal. The endurance test has something in common with its hardware equivalent, soak testing.

The bathtub curve. The name derives from the cross-sectional shape of a bathtub. Image source: wikipedia

This leads to another hardware concept which I think can be partially applied to software - the bathtub curve, as shown above. The bathtub curve is used in reliability engineering to describe a particular form of the hazard function which comprises three parts.

Applied to hardware, the bathtub curve means:

The first part is a decreasing failure rate, known as early failures. Burn in testing aims to detect (and discard) products which fail at this stage. If the burn-in period is made sufficiently long (and artificially stressful), the system can then be trusted to be mostly free of further early failures once the burn-in test is complete.
The second part is a constant failure rate, known as random failures. These are like the "background" level of failures, which can usually never be totally eliminated from the production process. QC managers will often aim for a low level of failure here, via various quality control measures (such as TQM or Six Sigma).
The third part is an increasing failure rate, known as wear-out failures. These can be detected via soak testing. In electronics, soak testing involves testing a system up to or above its maximum ratings for a long period of time. Often, a soak test can continue for months, while also applying additional stresses like extreme temperatures and/or pressures, depending on the intended environment.

Applied to software, the bathtub curve might show bug count on the y-axis and total application run-time on the x-axis. Then, the three parts could mean:

First part (early failures): In software, this could apply to bugs which break the software's functional requirements or specifications, and might be tested with various functional testing methods such as unit testing or integration testing. These are normally found early in the test cycle.
Second part (random failures): Might apply to random bugs which are hard to detect, occur in specific conditions, and are outside the existing test coverage. An example might be a heisenbug.
Third part (wear-out failures): Bugs which appear due to memory leaks or performance degredation. These can be tested with endurance testing. During endurance tests, memory utilization is monitored to detect potential memory leaks, and throughput/response times are monitored to detect performance degradation.

Further Reading

Since writing the above I've found some other articles (listed below) about this concept which assume time on the bathtub curve to mean the software development life cycle's time, as opposed to my idea of being the individual run-time of a real-time application. I've listed them here for reference:

The bathtub curve for software reliability
Does the bathtub curve apply to modern software?
The Software Bathtub Curve
Improving Software Reliability using Software Engineering Approach - A Review

Follow @dodgy_coder

Subscribe to posts via RSS

+ - - - - - - - - - - - - - - - - - - - - - - - +
| Harris Walker Real Estate, Perth, WA, AUS |
| Specialists in residential housing sales and |
| property management in Perth, Australia. |
+ - - - - - - - - - - - - - - - - - - - - - - - +

Why was .NET called .NET?

2012-05-05T22:56:00.000-07:00

Microsoft started development on the .NET Framework in the late 1990s, originally under the name of "Next Generation Windows Services" (NGWS).

So why did Microsoft choose the name .NET?

This is a bit of mystery, but the below answers are the best I've come across so far...

1) .NET enabled Microsoft's marketing people to emphasise the "Network"-ing aspect of its technologies, and was also a reaction to the marketing blitz by Sun Microsystems at the time, whose theme was "The network is the computer". The term "Dot Com" was synonymous with the Internet at the time, and "Dot Net" was a play on that term.
I don't think it is a bad name at all, the problem was that Microsoft initially named so many products with the ".NET" nomenclature like ".NET My Services" and ".NET Enterprise Servers", where the latter had nothing to do with the Internet. It caused so much confusion. Only later did Microsoft correct itself by limiting the .NET name to technologies related to the managed software framework.
- Stanley Siu

2) I was a dev at Microsoft at the time, and I have no idea whose ass the name .NET was pulled from. Anyone I talked to thought it was a lousy name for all the reasons already enumerated. At least it's pronounceable, unlike NGWS.
- George V. Reilly

3) The early marketing thrust of .NET was web services. .NET was supposed to make it easy both to write and consume web services. In particular, it was supposed to make it easier to call the web services that Microsoft was going to provide, and that everyone would then use: the ".NET My Services".
Of course, that fell apart very quickly, but the name remained. It was at least better than "COM++" or "ActiveXX".
- John Saunders

4) I was Summer intern at Microsoft in 2001 and back then the interns went to Bill Gates's house for a bbq near the end of the Summer. One of the interns asked "what other names did you think of before coming up with .NET?"

To the best of my memory, Bill's answer was something like:
"I didn't actually like the name .NET. It makes people wonder if we are finally just starting to learn about the Internet. Sadly, the other proposed name was even worse. Our mission statement at the time was 'work Anywhere, Anytime, on Any device,' so the proposal was AAAWare."
- leelin

Up until a couple of days ago, a Q&A on this topic was available here on StackOverflow, which is still the top Google result for: why was .net called .net.

The question was deemed to be offtopic for StackOverflow, and was recently deleted, even though it had been visible (as 'Closed') for about three years or so. Just before it was deleted, I had edited the accepted answer (by Stanley Siu) to make some grammatical corrections. Then, briefly after my edit was accepted by a moderator, the whole question was deleted. So I thought it would be worthy of this brief blog post here, in case someone else wants to find out the answer, as I did ;-).

UPDATE (7-May-2012): somehow (maybe due to a bit of attention from this post?), the original question on StackOverflow has been un-deleted and is once again available at its original location here. Thanks SO moderators!

Follow @dodgy_coder

Subscribe to posts via RSS

Programming language inventors on twitter

2012-04-29T04:54:00.000-07:00

Andrei Alexandrescu (D) - @incomputable
Joe Armstrong (Erlang) - @joeerl
Jeremy Ashkenas (CoffeeScript) - @jashkenas
Christopher Bertels (Fancy) - @bakkdoor
Henri Binsztok (Opa) - @henri_opa
Gilad Bracha (Newspeak) - @Gilad_Bracha
Walter Bright (D) - @WalterBright
Nicolas Cannasse (HaXe) - @ncannasse
John Carmack (QuakeC) - @ID_AA_Carmack
Alan Cooper (VisualBasic) - @MrAlanCooper
Rodrigo B. De Oliveira (Boo) - @nocastes
Steve Dekorte (Io) - @stevedekorte
Brendan Eich (JavaScript) - @BrendanEich
Andy Frank (Fantom) - @afrankvt
Brian Frank (Fantom) - @briansfrank
Rich Hickey (Clojure) - @richhickey
Graydon Hoare (Rust) - @graydon_moz
Stefan Karpinski (Julia) - @stefankarpinski
Guillaume Laforge (Groovy) - @glaforge
Rasmus Lerdorf (PHP) - @rasmus
Yukihiro Matsumoto (Ruby) - @yukihiro_matz
Bertrand Meyer (Eiffel) - @Bertrand_Meyer
Martin Odersky (Scala) - @odersky
Ray Ozzie (@Formula) - @rozzie
Slava Pestov (Factor) - @slava_pestov
Simon Peyton Jones (Haskell) - @simonpj0
Rob Pike (Go) - @rob_pike
Casey Reas (Processing) - @REAS
Carl Sassenrath (REBOL) - @rebol3
James Strachan (Groovy) - @jstrachan
Bjarne Stroustrup (C++) - @stroustrup
Don Syme (F#) - @dsyme
Tim Sweeney (UnrealScript) - @6ix7even
Guido van Rossum (Python) - @gvanrossum
Larry Wall (Perl) - @TimToady
John Warnock (PostScript) - @jewarnock
Stephen Wolfram (Mathematica) - @stephen_wolfram

If I've missed any others, please let me know via the comments at the bottom of this page, and I'll update the list.

I've added a twitter list with all of the above included here - twitter.com/dodgy_coder/prog-lang-inventors.

Programming Languages A-Z

Here's a list of programming languages in alphabetical order. Once again, if I've missed any significant ones, please let me know via the comments at the bottom and I'll update it.

ActionScript (1998)

Gary Grossman

A+ (1988)

Arthur Whitney

APL (1964)

Kenneth E. Iverson

Ada (1980)

Jean Ichbiah

AWK (1977)

Alfred Aho, Peter Weinberger, Brian Kernighan

BASIC(1964)

John George Kemeny, Thomas Eugene Kurtz

BCPL(1966)

Martin Richards

Boo(2003)

Rodrigo B. De Oliveira - @nocastes

C(1972)

Dennis Ritchie

C++(1983)

Bjarne Stroustrup - @stroustrup

C#(2001)

Anders Hejlsberg

Caml(1985)

Gérard Huet

Clojure(2006)

Rich Hickey - @richhickey

COBOL(1959)

Grace Hopper

Cobra(2008)

Chuck Esterbrook

CoffeeScript(2009)

Jeremy Ashkenas - @jashkenas

ColdFusion(1995)

D(2001)

Dart(2011)

Google - @dart_lang

Delphi(1996)

Eiffel(1986)

Bertrand Meyer - @Bertrand_Meyer

Erlang(1986)

Joe Armstrong - @joeerl

F# (2005)

Don Syme - @dsyme

Factor (2003)

Slava Pestov - @slava_pestov

Fancy(2010)

Christopher Bertels - @bakkdoor

Fantom (2005)

Brian Frank - @briansfrank
Andy Frank - @afrankvt

@Formula(1983)

Ray Ozzie - @rozzie

Forth(1973)

Charles H. Moore

FORTRAN(1957)

John Backus

Fortress(2007)

Guy L. Steele, Jr.

Go(2009)

Robert Griesemer
Rob Pike - @rob_pike
Ken Thompson

Groovy(2003)

Guillaume Laforge - @glaforge
James Strachan - @jstrachan

Haskell(1990)

Simon Peyton Jones - @simonpj0

HaXe(2005)

Nicolas Cannasse - @ncannasse

Inform(1993)

Graham Nelson

Io(2002)

Steve Dekorte - @stevedekorte

ISWIM(1966)

Peter J. Landin

Java(1995)

James Gosling

JavaScript(1995)

Brendan Eich - @BrendanEich

Joy(2000)

Manfred von Thun

Julia (2009)

Jeff Bezanson
Stefan Karpinski - @stefankarpinski
Viral Shah
Alan Edelman

Karel(1981)

Richard E. Pattis

Lisp(1958)

Logo(1967)

Lua(1993)

Roberto Ierusalimschy

Mathematica(1988)

Stephen Wolfram - @stephen_wolfram

MATLAB(1984)

Cleve Moler

Miranda(1985)

David Turner

ML(1973)

Robin Milner

Newspeak(2006)

Gilad Bracha - @Gilad_Bracha

Objective-C(1983)

Tom Love
Brad Cox

Octave(1988)

John W. Eaton

Opa (2011)

Henri Binsztok - @henri_opa

Oz(1991)

Gert Smolka

Pascal(1970)

Niklaus Wirth

Perl(1987)

Larry Wall - @TimToady

PHP(1995)

Rasmus Lerdorf - @rasmus

PostScript(1982)

Processing(2001)

Prolog(1972)

Alain Colmerauer

Pure (2008)

Albert Gräf

Python(1991)

Guido van Rossum - @gvanrossum

QuakeC(1996)

John Carmack - @ID_AA_Carmack

R(1993)

REBOL(1997)

Carl Sassenrath - @rebol3

Ruby(1995)

Yukihiro Matsumoto - @yukihiro_matz

Rust(2006)

Graydon Hoare - @graydon_moz

Scala(2003)

Martin Odersky - @odersky

Scheme(1975)

Simula(1967)

Smalltalk(1972)

SQL(1974)

Tcl(1988)

John Ousterhout

Tea(1997)

Jorge Nunes

UnrealScript(1998)

Tim Sweeney - @6ix7even

Verilog(1983)

Phil Moorby
Prabhu Goel

VisualBasic(1991)

Alan Cooper - @MrAlanCooper

WebDNA(1995)

Grant Hulbert

X10(2004)

Kemal Ebcioglu
Vijay Saraswat
Vivek Sarkar

Yorick(1996)

David H. Munro

ZPL(1993)

Brad Chamberlain
Sung-Eun Choi
E Christopher Lewis
Calvin Lin
Jason Secosky
Larry Snyder
W. Derrick Weathersby

Follow @dodgy_coder

Subscribe to posts via RSS

Scalability lessons from Google, YouTube, Twitter, Amazon, eBay, Facebook and Instagram

2012-04-09T02:41:00.000-07:00

I've gathered together in one place a few lessons in scalability from seven of the most highly trafficked websites around. I've grabbed this primarily from various articles on the excellent High Scalability Blog and have summarized the main points from each company below.

Here are some common ideas I've noticed across all seven companies ...

Keep it simple - complexity will come naturally over time.
Automate everything, including failure recovery.
Iterate your solutions - be prepared to throw away a working component when you want to scale it up to the next level.
Use the right tool for the job, but don't be afraid to roll your own solution.
Use caching, where appropriate.
Know when to favor data consistency over data availability, and vice versa.

Google

Reliable storage
Reliable scalable storage is a core need of any application. The Google File System (GFS) is Google's core storage platform - its a large distributed log structured file system into which they throw a lot of data. Why did they build it instead of using something off the shelf? Because they control everything about it, and it's the platform that distinguishes them from everyone else. From GFS, they gain: high reliability across data centers, scalability to thousands of network nodes, huge read/write bandwidth, support for large blocks of data which are gigabytes in size and an efficient distribution of operations across nodes to reduce bottlenecks.

Infrastructure as a competitive advantage
Google can roll out new internet services faster, cheaper, and at scale at few others can compete with. Many companies take a completely different approach by treating infrastructure as an expense. Each group will use completely different technologies and there will be little planning and commonality of how to build systems.

Build applications on top of a platform
An under appreciated advantage of a platform approach is that junior developers can quickly and confidently create robust applications. If every project needs to create the same distributed infrastructure you'll quickly run into difficulty because the people who know how to do this are relatively rare. Synergy isn't always crap. By making all parts of a system work together an improvement in one helps them all. Improve the file system and everyone benefits immediately and transparently. If every project uses a different file system then there's no continual incremental improvement across the entire stack.

Automation and recovery
Build self-managing systems that work without having to take the system down. This allows you to more easily rebalance resources across servers, add more capacity dynamically, bring machines off line, and gracefully handle upgrades.

Create a Darwinian infrastructure
Perform a time consuming (CPU bound) operation in parallel and take the winner. This is especially useful when you have spare CPU capacity but limited IO.

Don't ignore the Academy
Academia has a lot of good ideas that don't get translated into production environments. Most of what Google has done has prior art, just not prior large scale deployment.

Consider data compression
Compression is a good option when you have a lot of CPU to throw around and limited IO.

YouTube

Keep it simple
Look for the simplest thing that will address the problem space. There are lots of complex problems, but the first solution doesn’t need to be complicated. The complexity will come naturally over time. Choose the simplest solution possible with the loosest guarantees that are practical. The reason you want all these things is you need flexibility to solve problems. The minute you over specify something you paint yourself into a corner. You aren’t going to make those guarantees. Your problem becomes automatically more complex when you try and make guarantees - you leave yourself no way out.

Cheat: know how to fake data
The fastest function call is the one that doesn’t happen. When you have a consistently increasing counter, like a view count, you would need to do a database call every update. Or you could do a call every once in awhile, and update by a random amount in between - people will believe it’s real. Know how to fake data.

Jitter: add entropy back into your system
If your system doesn’t jitter then you get thundering herds of people all requesting the same resource at the same time. For a popular video, they cache things as best they can. The most popular video they might cache for 24 hours. If everything expires at the same time, then every machine will calculate the expiration at the same time. This creates a thundering herd. By jittering you are saying randomly expire between 18-30 hours. That prevents things from happening at the same time and spreads requests out over a long period of time.

Approximate correctness
The state of the system is that which the user sees. If a user can’t tell a part of the system is skewing and inconsistent, then it’s not. If you write a comment and someone loads the page at the same time, they might not get it for half a second, the user who is just reading the page won’t care. The writer of the comment will care though, so you make sure the user who wrote the comment will see it immediately. This allows you to cheat a little bit. When it comes to comments, your system doesn’t have to have globally consistent transactions. That would be super expensive and overkill. Comments are not financial transaction - know when you can cheat.

Twitter

Implement an API
Twitter's API Traffic is ten times that of Twitter’s Website alone. The API is the most important thing Twitter did to grow their user base. Keeping the service simple allowed developers to build on top of their infrastructure and come up with app ideas that are way better than Twitter could come up with. You can never do all the work your user's can do and you probably won't be as creative. So open up your application and make it as easy as possible for others to integrate your application with theirs.

Use what you know
Twitter uses messaging a lot. Producers produce messages, which are queued, and then are distributed to consumers. Twitter's main functionality is to act as a messaging bridge between different formats (SMS, web, IM, etc). Send a message to invalidate a friend's cache in the background instead of doing it all individually, synchronously. Twitter developers were most familiar with Ruby, so they moved from DRb to Starling, a distributed queue written in Ruby. Distributed queues were made to survive system crashes by writing them to disk. In Twitter's experience, most performance improvements come not from the choice of language, but from an application's design.

Know when and what to cache
For example, getting your friends status is complicated. There are security and other issues. So rather than doing a database query, a friend's status is updated in cache instead. It never touches the database. 90% of requests are API requests. So they don't do any page caching for the front-end. Twitter pages are so time sensitive it doesn't do any good. Twitter only caches API requests.

Defend yourself against abuse
Understand how people will try to bust your system. Put in reasonable limits and detection mechanisms to protect your system from being killed by bots. Build tools to detect abuse so you can pinpoint when and where they are happening. Be ruthless. Delete them as users.

Amazon

Use SOA
Amazon's architecture is loosely coupled and built around services. A service-oriented architecture (SOA) gave them the isolation that would allow building many software components rapidly and independently of each other, allowing fast time to market. The application that renders the Amazon.com Web pages is one such application server. So are the applications that serve the Web-services interface, the customer service application, and the seller interface.

Open up your system with APIs and you'll create an ecosystem around your application. Organizing around services gives you agility - you can now do things in parallel is because the output of everything is a service. Prohibit direct database access by clients. This means you can make you service scale and be more reliable without involving your clients. This is much like Google's ability to independently distribute improvements in their stack to the benefit of all applications.

Know when to favour consistency over availability and vice versa
To scale you have to partition, so you are left with choosing either high consistency or high availability for a particular system. You must find the right overlap of availability and consistency. Choose a specific approach based on the needs of the service. For the checkout process you always want to honor requests to add items to a shopping cart because it's revenue producing. In this case you choose high availability. Errors are hidden from the customer and sorted out later. When a customer submits an order you favor consistency because several services (credit card processing, shipping and handling, reporting) are simultaneously accessing the data and each rely on their data to be consistent.

Embrace failure
Take it for granted stuff fails, that's reality, embrace it. For example, go more with a fast reboot and fast recover approach. With a decent spread of data and services you might get close to 100%. Create a self-healing, self-organizing, lights-out type operation.

Only use what you need
Keep things simple by making sure there are no hidden requirements and hidden dependencies in the design. Cut technology to the minimum you need to solve the problem you have. It doesn't help the company to create artificial and unneeded layers of complexity. Not stuck with one particular approach or technology stack. Some places they use jboss/java, but they use only servlets, not the rest of the J2EE stack. C++ is uses to process requests. Perl/Mason is used to build content.

Base decisions on customer feedback
Use measurement and objective debate to separate the good from the bad. Expose real customers to a choice and see which one works best and to make decisions based on those tests. This is done with techniques like A/B testing and Web Analytics. If you have a question about what you should do - code it up, let people use it, and see which alternative gives you the results you want.

Scalability as a competitive advantage
Infrastructure for Amazon, like for Google, is a huge competitive advantage. They can build very complex applications out of primitive services that are by themselves relatively simple. They can scale their operation independently, maintain unparalleled system availability, and introduce new services quickly without the need for massive reconfiguration.

eBay

Partition everything
If you can't split it, you can't scale it. Split everything into manageable chunks by function and data.

Asynchrony everywhere
Connect independent components through event-driven queues and pipelines.

Embrace failure
Monitor everything, provide service even when parts start failing. Minimize and control dependencies, use abstract interfaces and virtualization, components have an SLA, consumers responsible for recovering from SLA violations. Automate everything. Components should automatically adjust and the system should learn and improve itself.

Embrace inconsistency
Pick for each feature where you need to be on the CAP continuum, no distributed transactions, inconsistency can be minimized by careful operation ordering, become eventually consistent through async recovery and reconciliation.

Save all your data
Data drives finding optimization opportunities, predictions, recommendations, so save it all. Know which data is authoritative, which data isn't, and treat it accordingly.

Infrastructure: use the right tool for the right job
Need to maximize utilization of every resource: data (memory), processing (CPU), clock time (latency), power. One size rarely fits all, particularly at scale. Compose from orthogonal, commodity components.

Facebook

Scaling takes multiple iterations
Solutions often work in the beginning, but you'll have to modify them as you go on - what works in year one may not work later. A good example is photos. Facebook currently serves 1.2 million photos a second. The first generation was built the easy way - don't worry about scaling that much - focus on getting the functionality right. The uploader stored the file in NFS and the meta-data was stored in MySQL. It only worked for the first 3 months but this didn't matter because time to market was the biggest competitive advantage they had. Having the feature was more important than making sure it was a fully thought out, scalable solution. The second generation was optimized - different access patterns were optimized for. Smaller images were accessed more frequently so those became cached. They also started using a CDN (Content Delivery Network). The third generation is an overlay system that creates a file that is a blob stored in the file system. Images are stored in a binary blob and you know the byte offset of the photo in the blob - so there's only one disk IO per photo.

Don't over design a solution - keep it simple
Just use what you need to use as you scale your system out. Figure out where you need to iterate on a solution, optimize something, or completely build a part of the stack yourself. Facebook spent a lot of time trying to optimize PHP, and ended up writing HipHop, a tool to convert PHP into C++. This generated a massive amount of memory and CPU savings. You don't have to do this on day one, but you may have to. Focus on the product first before you write an entire new language.

Choose the right tool for the job, but accept that your choice comes with overhead
If you really need to use Python then go ahead and do so, but realize with that choice there is overhead, usually across deployment, monitoring and operations. If you choose to use a service oriented architecture (SOA) you'll have to build most of the backend yourself and that often takes quite a bit of time. With the LAMP stack you get a lot for free. Once you move away for the LAMP stack, how you do things like service configuration and monitoring is up to you. As you go deeper into the services approach you have to reinvent the wheel.

Get the culture right
Build an environment internally which promotes building the right thing first and then fixing as needed, not worrying about innovating, not worrying about breaking things, thinking big, thinking what is the next thing you need to build after the building the first thing. You can get the code right, you can get the products right, but you need to get the culture right first. If you don't get the culture right then your company won't scale.

Instagram

Make use of existing cloud infrastructure
Don't reinvent the wheel when you can use solid and proven technology. Instagram runs 100+ instances of Ubuntu 11.04 on Amazon's EC2 cloud computing infrastructure. They also use Amazon's ELB (Elastic Load Balancer), which comprises three NGINX instances, with automatic failure recovery. The photos themselves go straight to Amazon S3 storage and they use Amazon CloudFront as their CDN (Content Delivery Network), which helps with image load times from users around the world (like in Japan, their second most-popular country).

Asynchronous task queuing
When a user decides to share out an Instagram photo to Twitter or Facebook, or when they need to notify a Real-time subscriber of a new photo posted, they push the task into the open source Gearman task management framework. Doing it asynchronously through a task queue means that media uploads can finish quickly, while the ‘heavy lifting’ can run in the background. There's about 200 workers, written in Python, consuming the task queue at any given time, split between the services they share to.

Push notifications
They use an open source Apple Push Notification Service (APNS) provider called pyapns, which is based on Twisted. It has handled over a billion push notifications for Instagram, and they report that its been rock-solid.

Real-time system-wide monitoring
With 100+ EC2 instances, it’s important to keep on top of what’s going on across the board. They use Munin to graph metrics system-wide, which sends an alert if anything is outside of its normal operating range. They write custom Munin plugins, building on top of Python-Munin, to graph metrics that aren’t system-level (for example, signups per minute, photos posted per second, etc). They use Pingdom for external monitoring of the service, and PagerDuty for handling notifications and incidents. For Python error reporting, they use Sentry, an open-source Django app. At any given time, they can sign-on and see what errors are happening across their system, in real time.

Selective use of NoSQL technology (like Redis)
Redis powers the main feed, the activity feed, the sessions system (here’s their Django session backend), and other related systems. All of Redis’ data needs to fit in memory, so they run several Quadruple Extra-Large Memory instances on EC2 for Redis, and occasionally shard across a few Redis instances for any given subsystem.

Sidenote on CAP

Eric Brewer's CAP Theorem or "the three properties of systems"
There are three properties of a system: consistency, availability and tolerance to network partitions (partitionability).

You can have at most two of these three properties for any shared-data system.

Consistency: write a value and then when you read the value you get the same value back. In a partitioned system there are time windows where that's not true.

Availability: you may not always be able to write or read. The system may prevent you from writing because it wants to keep the system consistent.

Partitionability: divide nodes into small groups that can see other groups, but they can't see everyone.

Follow @dodgy_coder

Subscribe to posts via RSS

Every software project I’ve worked on has used the "Spanish Theory" of project management, and its likely yours have too

2012-03-03T03:58:00.000-08:00

The "Spanish Theory" says that management's job is to extract the maximum resources (= developer effort) from the smallest amount of money (= developer salary). In practice what this often means for the developer is unpaid overtime (also known as "crunch time"), something very familiar to game developers, and also common in traditional software development, as the project nears its deadline. But those unpaid hours are actually costing you, the developer, because you can't get them back. You've sacrificed time in your personal life with your family and instead have chosen to work on the company's project - something of large value has been sacrificed for something of lesser value. If this imbalance continues past a reasonable level and unpaid overtime becomes the norm, then many developers will become dissatisfied and leave the company, increasing the company's staff turnover (churn) rate.

The book describing these harmful effects in the software industry was first published back in 1987. So why are many software development managers and PMs still practicing this method, some 25 years later? The reason I believe is a lack of foresight, an unquestioning acceptance of previous methods, a short term project oriented view, a limited understanding of the harm that comes to a company from a high churn rate, and finally, developers that are willing to accept these conditions. In moderation, some unpaid overtime is probably ok, but when it becomes the norm, from week to week and is just expected of all developers, then maybe that's a sign something is wrong, and could even be considered a job smell.

The attitude of government and corporations around the world towards unpaid overtime is changing, as can be seen by some recent initiatives out of Britain (Work your proper hours day). Similarly in Australia, there's now a National go home on time day held every November. The US-based International Game Developers Association (IGDA) is also active in promoting quality of life in the workplace through various publications and events.

Below I've quoted the extracts from chapter 3 of the book which gave me the idea for this brief blog post - I highly recommend you have a read of this book, or if not, then at least read through this small extract here which I thought was outstanding ...

Peopleware: Productive Projects and Teams (Second Edition)
Amazon Link: here
Authors: Tom Demarco and Timothy Lister

"Historians long ago formed an abstraction about different theories of value: The Spanish Theory, for one, held that only a fixed amount of value existed on earth, and therefore the path to the accumulation of wealth was to learn to extract it more efficiently from the soil or from people's backs.

Then there was the English Theory that held that value could be created through ingenuity and technology. So the English had an Industrial Revolution, while the Spanish spun their wheels trying to exploit the land and the Indians in the New World. They moved huge quantities of gold across the ocean, and all they got for their effort was enormous inflation (too much gold money chasing too few usable goods).

The Spanish Theory of Value is alive and well among managers everywhere. You see that whenever they talk about productivity. Productivity ought to mean achieving more in an hour of work, but all too often it has come to mean extracting more for an hour of pay. There is a large difference. The Spanish Theory managers dream of attaining new productivity levels through the simple mechanism of unpaid overtime. They divide whatever work is done in a week by forty hours, not by the eighty or ninety hours that the worker actually put in.

That's not exactly productivity — it's more like fraud — but it's the state of the art for many American managers. They bully and cajole their people into long hours. They impress upon them how important the delivery date is (even though it may be totally arbitrary; the world isn't going to stop just because a project completes a month late). They trick them into accepting hopelessly tight schedules, shame them into sacrificing any and all to meet the deadline, and do anything to get them to work longer and harder.

Overtime for salaried workers is a figment of the naive manager's imagination. Oh, there might be some benefit in a few extra hours worked on Saturday to meet a Monday deadline, but that's almost always followed by an equal period of compensatory "undertime" while the workers catch up with their lives. Throughout the effort there will be more or less an hour of undertime for every hour of overtime. The trade-off might work to your advantage for the short term, but for the long term it will cancel out.

Just as the unpaid overtime was largely invisible to the Spanish Theory manager (who always counts the week as forty hours regardless of how much time the people put in), so too is the undertime invisible. You never see it on anybody's time sheet. It's time spent on the phone or in bull sessions or just resting. Nobody can really work much more than forty hours, at least not continually and with the level of intensity required for creative work. Overtime is like sprinting: It makes some sense for the last hundred yards of the marathon for those with any energy left, but if you start sprinting in the first mile, you're just wasting time. Trying to get people to sprint too much can only result in loss of respect for the manager. The best workers have been through it all before; they know enough to keep silent and roll their eyes while the manager raves on that the job has got to get done by April. Then they take their compensatory undertime when they can, and end up putting in forty hours of real work each week. The best workers react that way; the others are workaholics.

People under time pressure don't work better; they just work faster.

In order to work faster, they may have to sacrifice the quality of the product and their own job satisfaction."

Follow @dodgy_coder

Subscribe to posts via RSS

Coding tricks of game developers

2012-02-11T01:45:00.000-08:00

If you've got any real world programming experience then no doubt at some point you've had to resort to some quick and dirty fix to get a problem solved or a feature implemented while a deadline loomed large. Game developers often experience a horrific "crunch" (also known as a "death march"), which happens in the last few months of a project leading up to the game's release date. Failing to meet the deadline can often mean the project gets cancelled or even worse, you lose your job. So what sort of tricks do they use while they're under the pump, doing 12+ hour per day for weeks on end?

Below are some classic anecdotes and tips - many thanks to Brandon Sheffield who originally put together this article on Gamasutra. I have reposted a few of his stories and also added some more from newer sources. I have also linked on each story to the author's home page or blog wherever possible.

1. The programming antihero - Noel Llopis

I was fresh out of college, still wet behind the ears, and about to enter the beta phase of my first professional game project -- a late-90s PC title. It had been an exciting rollercoaster ride, as projects often are. All the content was in and the game was looking good. There was one problem though: We were way over our memory budget.

Since most memory was taken up by models and textures, we worked with the artists to reduce the memory footprint of the game as much as possible. We scaled down images, decimated models, and compressed textures. Sometimes we did this with the support of the artists, and sometimes over their dead bodies.

We cut megabyte after megabyte, and after a few days of frantic activity, we reached a point where we felt there was nothing else we could do. Unless we cut some major content, there was no way we could free up any more memory. Exhausted, we evaluated our current memory usage. We were still 1.5 MB over the memory limit!

At this point one of the most experienced programmers in the team, one who had survived many years of development in the "good old days," decided to take matters into his own hands. He called me into his office, and we set out upon what I imagined would be another exhausting session of freeing up memory.

Instead, he brought up a source file and pointed to this line:

static char buffer[1024*1024*2];

"See this?" he said. And then deleted it with a single keystroke. Done!

He probably saw the horror in my eyes, so he explained to me that he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to free up when it's really needed.

He walked out of the office and announced he had reduced the memory footprint to within budget constraints -- he was toasted as the hero of the project.

As horrified as I was back then about such a "barbaric" practice, I have to admit that I'm warming up to it. I haven't gotten into the frame of mind where I can put it to use yet, but I can see how sometimes, when you're up against the wall, having a bit of memory tucked away for a rainy day can really make a difference. Funny how time and experience changes everything.

2. Cache it up - Andrew Russell

To improve performance when you are processing things in a tight loop, you want to make the data for each iteration as small as possible, and as close together as possible in memory. That means the ideal is an array or vector of objects (not pointers) that contain only the data necessary for the calculation.

This way, when the CPU fetches the data for the first iteration of your loop, the next several iterations worth of data will get loaded into the cache with it.

There's not really much you can do with using fewer and faster instructions because the CPU is as fast as its going to get, and the compiler can't be improved. Cache coherence is where it's at - this article contains a good example of getting cache coherency for an algorithm that doesn't simply run through data linearly.

3. Plan your distractions - Jay Barnson

The Internet is one of the greatest tools ever invented for both improving and destroying productivity. Twitter and forums and blogs and instructional websites can be extremely motivational and educational, but they can also be a distraction that completely destroys all hope of ever getting anything done. One thing I’ve done in the past which has proven pretty successful is to stick to a plan for when I can spend some minutes checking email and Twitter, or play a quick game or something. Either at the completion of a task, or after a period of time (say one five-minute break every hour). Otherwise, the browser’s only use is for reading reference manual pages, if necessary. That way I turn a potential distraction into a motivating tool.

4. Collateral damage - Jim Van Verth (@cthulhim)

Don't know how many remember Force 21, but it was an early 3D RTS which used a follow cam to observe your current platoon. Towards the end of the project we had a strange bug where the camera would stop following the platoon -- it would just stay where it was while your platoon moved on and nothing would budge it. The apparent cause was random because we couldn't find a decent repro case. Until, finally, one of the testers noticed that it happened more often when an air strike occurred near your vehicles. Using that info I was able to track it down.

Because the camera was using velocity and acceleration and was collidable, I derived it from our PhysicalObject class, which had those characteristics. It also had another characteristic: PhysicalObjects could take damage. The air strikes did enough damage in a large enough radius that they were quite literally "killing" the camera.

I did fix the bug by ensuring that cameras couldn't take damage, but just to be sure, I boosted their armor and hit points to ridiculous levels. I believe I can safely say we had the toughest camera in any game.

5. The blind leading the blind - Maurício Gomes

At university there was a team that made a FPS flash game. For some bizarre reason, the programmer, instead of checking if the character was colliding with the wall and stop you going there, he did the inverse, he checked if there was a wall, and only allowed you to move parallel to it!

This sparked a bizarre bug: in crossings or T junctions in the level, you could not actually cross, only turn to the passage on your left or right. The deadline was closing, and they had no idea on how to fix it.

Then the team writer fixed the issue; he told the artist to add an animation of hands touching the walls, and then he added in the background story that the main character was blind and needed to constantly touch the walls to know where he was going.

6. You wouldn't like me when I'm angry - Nick Waanders

I once worked at THQ studio Relic Entertainment on The Outfit, which some may remember as one of the earlier games for the Xbox 360. We started with a PC engine (single-threaded), and we had to convert it to a complete game on a next-gen multi-core console in about 18 months. About three months before shipping, we were still running at about 5 FPS on the 360. Obviously this game needed some severe optimization.

When I did some performance measurements, it became clear that as much as the code was slow and very "PC," there were also lots of problems on the content side as well. Some models were too detailed, some shaders were too expensive, and some missions simply had too many guys running around.

It's hard to convince a team of 100 people that the programmers can't simply "fix" the performance of the engine, and that some of the ways people had gotten used to working to needed to be changed. People needed to understand that the performance of the game was everybody's problem, and I figured the best way to do this is with a bit of humor that had a bit of hidden truth behind it.

The solution took maybe an hour. A fellow programmer took four pictures of my face -- one really happy, one normal, one a bit angry, and one where I am pulling my hair out. I put this image in the corner of the screen, and it was linked to the frame rate. If the game ran at over 30fps, I was really happy, if it ran below 20, I was angry.

After this change, the whole FPS issue transformed from, "Ah, the programmers will fix it." to, "Hmm, if I put this model in, Nick is going to be angry! I'd better optimize this a little first." People could instantly see if a change they made had an impact on the frame rate, and we ended up shipping the game at 30fps.

7. Its not a bug, its a feature! - Philip Tan

I worked on an RPG in which we were trying to get the NPCs (Non-player Characters) to spot when you were in range, walk up to you, and strike up a conversation with you by activating the dialog system.

We forgot to add code to distinguish NPCs from PCs (Player Characters), so we'd walk into town and all the NPCs would be talking with each other. Because all NPC AI code used the same dialog template, they actually got a few sentences in before the conversations became nonsensical. And because character dialog was broadcast, you could read everything they said if you were in range.

We decided to turn that bug into a major feature.

8. Dirty deeds - Tim Randall (Developer @ Encore)

The engine team at Gremlin Interactive used to keep a single glove in their office. When someone asked why it was there, they were told it was only used when someone was about to type some really dirty code. It wasn't so much a case of not wanting to leave fingerprints but rather not wanting to actually touch the dirtiest fixes!

9. Explicit conditional hinting - ZorbaTHut

A very, very low-level tip, but one that can come in handy... most compilers support some form of explicit conditional hinting. GCC has a function called __builtin_expect which lets you inform the compiler what the value of a result probably is. GCC can use that data to optimize conditionals to perform as quickly as possible in the expected case, with slightly slower execution in the unexpected case.

if(__builtin_expect(entity->extremely_unlikely_flag, 0)) {
// code that is rarely run
}

I've seen a 10-20% speedup with proper use of this.

10. Object-ive oriented programming - Anonymous

Back at a game studio, I think it was near the end of the project, we had an object in one of the levels that needed to be hidden. We didn't want to re-export the level and we did not use checksum names. So right smack in the middle of the engine code we had something like the following:

if( level == 10 && object == 56 )
{
HideObject();
}

The game shipped with this in.

Maybe a year later, an artist using our engine came to us very frustrated about why an object in their level was not showing up after exporting. The level they had a problem with resolved to level 10. I wonder why?

11. Stack vs Heap - Torbjörn Gyllebring

Stack allocation is much faster than heap allocation since all it really does is move the stack pointer. Using memory pools, you can get comparable performance out of heap allocation, but that comes with a slight added complexity and its own headaches.

Also, stack vs heap is not only a performance consideration; it also tells you a lot about the expected lifetime of objects. The stack is always hot, the memory you get is much more likely to be in cache than any far heap allocated memory.

Downside of the stack is that it is actually a stack. You can't free a chunk of memory used by the stack unless it is on top of it. There's no management, you push or pop things on it. On the other hand, the heap memory is managed: it asks the kernel for memory chunks, maybe splits them, merges thems, reuses them and frees them. The stack is really meant for fast and short allocations.

12. I'm a programmer, not an artist - Damian Connolly

For indie / solo developers who are working on an iPhone or Android game on their own, while you're looking for an artist etc, you should be developing your game at the same time. Use programmer art, stand-ins, free sprites anything. Most of the time, before even thinking about final assets, I just want something up and running quickly to see if it's fun. Prototype the crap out of it and find the game. Then, when the gameplay's locked down, you can start putting in the proper art. Doing it the other way around leads to lost money, and work that needs to be redone multiple times, which aside from harming your project, sucks your motivation to finish it (and if you're making a game to get a job, showing that you can finish a project is a good thing). Another tip if you're lacking upfront finance is to find a freelance game artist who will accept a revenue sharing deal, e.g. typically something like 30% of game revenue, payable once it gets published to the AppStore.

13. Remove unnecessary branches - tenpn

On some platforms and with some compilers, branches can throw away your whole pipeline, so even insignificant if() blocks can be expensive.

The PowerPC architecture (PS3/x360) offers the floating-point select instruction, fsel. This can be used in the place of a branch if the blocks are simple assignments:

float result = 0;
if (foo > bar) { result = 2.0f; }
else { result = 1.0f; }

Becomes:

float result = fsel(foo-bar, 2.0f, 1.0f);

When the first parameter is greater than or equal to 0, the second parameter is returned, else the third. The price of losing the branch is that both the if{} and the else{} block will be executed, so if one is an expensive operation or dereferences a NULL pointer this optimisation is not suitable. Sometimes your compiler has already done this work, so check your assembly first.

14. Hack the stack - Steve DeFrisco

I was one of a few interns at IMAGIC in 1982-83. We were all doing Intellivision carts. One of the programmers had to leave to go back to school, and I was chosen to fix the random crash bug in his game. It turned out to be a stack overflow in the timer interrupt handler. Since the only reason for the handler was to update the *display* of the on-screen timer, I added some code to test the depth of the stack at the beginning of the interrupt routine. If we were in danger of overflowing the stack, return without doing anything. Since the handler was called multiple times per second, the player never noticed, and the crash was fixed.

15. Meet my dog, "Patches" - Mick West

There's an old joke that goes something like this:

Patient: "Doctor, it hurts when I do this."
Doctor: "Then stop doing it."

Funny, but are these also wise words when applied to fixing bugs? Consider the load of pain I found myself in when working on the port of a 3D third person shooter from the PC to the original PlayStation.

Now, the PS1 has no support for floating point numbers, so we were doing the conversion by basically recompiling the PC code and overloading all floats with fixed point. That actually worked fairly well, but where it fell apart was during collision detection.

The level geometry that was supplied to us worked reasonably well in the PC version of the game, but when converted to fixed point, all kinds of seams, T-Junctions and other problems were nudged into existence by the microscopic differences in values between fixed and floats. This problem would manifest itself in one case with the main character touching a particular type of door in a particular level in a particular location; rather than fix the root cause of the problem, I simply made it so that if he ever touched the door, then I'd move him away, and pretend it never happened. Problem solved.

Looking back I find this code quite horrifying. It was patching bugs and not fixing them. Unfortunately the real fix would have been to go and rework the entire game's geometry and collision system specifically with the PS1 fixed point limitations in mind. The schedule was initially aggressive, and since we always seemed close to finishing, the quick patch option won over against a comprehensive (but expensive) fix.

But it did not go well. Hundreds of patches were needed, and then the patches themselves started causing problems, so more patches were added to turn off the patches in hyper-specific circumstances. The bugs kept coming, and I kept beating them back with patches. Eventually I won, but at a cost of shipping several months behind schedule, and working 14 hour days for all of those several months.

That experience soured me against "the patch." Now I always try to dig right down to the root cause of a bug, even if a simple, and seemingly safe, patch is available. I want my code to be healthy. If you go to the doctor and tell him "it hurts when I do this," then you expect him to find out why it hurts, and to fix that. Your pain and your code's bugs might be symptoms of something far more serious. The moral: Treat your code like you would want a doctor to treat you; fix the cause, not the symptoms.

16. Identity crisis - Noel Llopis

This scene is familiar to all game developers: It's the day we're sending out the gold candidate for our Xbox 1 game. The whole team is playtesting the game all day long, making sure everything looks good. It's fun, it's solid, it's definitely a go in our minds.

In the afternoon, we make the last build with the last few game-balancing tweaks, and do one last playthrough session when disaster strikes: The game crashes hard! We all run to our workstations, fire up the debugger, and try to figure out what's going on. It's not something trivial, like an assert, or even something moderately hard to track down, like a divide by zero. It looks like memory is garbage in a few places, but the memory reporting comes out clean. What's going on?

One dinner and many hours later, our dreams of getting out on time shattered, we manage to track it down to one data file being loaded in with the wrong data. The wrong data? How's that possible? Our resource system boiled down every asset to a 64-bit identifier made out of the CRC32 of the full filename and the CRC32 of all the data contents. That was also our way of collapsing identical resource files into a single one in the game. With tens of thousands of files, and two years of development, we never had a conflict. Never.

Until now, that is.

It turns out that one of the innocent tweaks the designers had checked in that afternoon made it so a text file had the exact same filename and data CRC as another resource file, even though they were completely different!

Our hearts sank to our feet when we recognized the problem. There's no way we could change the resource indexing system in such a short period of time. Even if we pulled an all-nighter, there was no way to know for sure that everything would be stable in the morning.

Then, as quickly as despair swept over us, we realized how we could fix this on time for the gold candidate release. We opened up the text file responsible for the conflict, added a space at the end, and saved it. We looked at each other with huge grins on our faces and said:

"Ship it!"

The extra space meant the CRC32 checksum of the text file was altered and therefore no longer conflicted with the other resource.

17. HexEdit to the rescue - Ken Demarest

Back on Wing Commander 1 we were getting an exception from our EMM386 memory manager when we exited the game. We'd clear the screen and a single line would print out, something like "EMM386 Memory manager error. Blah blah blah." We had to ship ASAP. So I hex edited the error in the memory manager itself to read "Thank you for playing Wing Commander."

18. 8-bit audio stomper - Toonse

For a launch product of a certain console I had a nasty bug report from QA that took 20+ hours to reproduce. Finally (with 24 hours left to go to hit console launch) tracked it down to some audio drivers in the firmware that were erroneously writing 1 random byte "somewhere" at random times where the "somewhere" was always in executable code space. I finally figured out that any given run of the game that "somewhere" was always the same place, luckily. 1st party said sorry, can't fix it in time as we don't know why it's being caused! So I shipped that game with stub code at the very start of main that immediately saved off the 1 byte from the freshly loaded executable in the place I knew it would overwrite for that particular version of the exe. There was then code that would run each frame after audio had run and restore that byte back to what it should be just in case it had been stomped that frame. Good times! We hit launch.

To this day I still feel very very dirty about this hack, but it was needed to achieve the objectives and harmed no-one :)

19. Rainy day server pool - Potatolicious

I used to work for a company that had a horrific hardware requisition policy. If your team needed a server, it had to go through a lengthy and annoying approvals process - and even then, it took months before Infrastructure would actually provide said servers.

In other words, when a project gets handed down from above to launch in, say, 3 months, there's no way in hell you can get the servers requisitioned, approved, and installed in that time. It became standard practice for each team to slightly over-request server capacity with each project and throwing the excess hosts into a rainy day pool, immediately available and repurposeable as required.

New servers will still get requested for these projects, but since they took so long to approve, odds are they'd go right into the pool whenever they actually arrived, which sometimes took up to a year.

Of course, it was horrifyingly inefficient. Just on my team alone I think we had easily 50 boxes sitting around doing nothing (and powered on to boot) waiting to pick up the slack of a horrendously broken bureaucracy.

20. Bit shifting magic - Steven Pigeon

In order to avoid stalls in the processor pipeline due to branching, one can often use a branchless equivalent, that is, code transformed to remove the if-then-elses and therefore jump prediction uncertainties. E.g. a straightforward implementation of abs( ) in C might be

inline int abs(int x)
{
return (x<0) ? -x : x;
}

Which is simple enough but contains an inline if-then-else. As the argument, x, isn’t all that likely to follow a pattern that the branch prediction unit can detect, this simple function becomes potentially costly as the jump will be mispredicted quite often.

How can we remove the if-then-else, then? One solution is to use the right shift operator (>>) and the bitwise XOR operator (^) as following:

inline int abs_no_branch(int x)
{
int m = (x >> (8 * sizeof(int)-1));
return ((x ^ m) - m);
}

Where the expression (8 * sizeof(int) - 1) evaluates to 15, 31, or 63 depending on the size of integers on the target computer.

Further detailed reading about this technique here: Branchless Equivalents of Simple Functions.

Follow @dodgy_coder

Subscribe to posts via RSS

The Promise of HTML5

2012-01-15T02:26:00.000-08:00

Anyone wanting evidence of what HTML5 can achieve needs only to browse through the many excellent demos around the place, such as at chromeexperiments.com; these run inside WebGL enabled browsers, such as Google Chrome and Firefox. One of the driving forces of HTML5 is to make it easy to include and handle multimedia and graphical content on the web without having to resort to proprietary plugins and APIs; the new <canvas>, <svg>, <video>, <audio> tags allow for much of this. WebGL is a context of the canvas HTML element that provides a 3D computer graphics API without the use of plug-ins. WebGL is currently supported by all major desktop browsers, except for Internet Explorer where it is currently only supported via a plugin (named IEWebGL). Support on mobile devices is a bit patchy though, with it being limited to Mobile Firefox on Android along with some specific Nokia, Sony Ericsson and Blackberry models. On the iPhone, there is a workaround / hack to develop apps with WebGL, and its expected Apple will support it in a future version.

This demo by Evan Wallace shows off the power of WebGL - a pool of water rendered with reflection, refraction, caustics, and ambient occlusion. You can create ripples on the water and also move the sphere around, both in and out of the water.

A popular lightweight framework has sprung up on top of WebGL called three.js.The aim of the project is to create a lightweight 3D engine with a very low level of complexity; in other words, for developers who are not experienced with 3D graphics coding. The engine can render using <canvas>, <svg> and also WebGL. It is one of the most comprehensive WebGL frameworks, covering matrices, scenes, meshes, materials, shadows, voxels, fog, and more. Many of the Chrome WebGL demos linked above use it and there's a list of demos on the three.js github page. These demos are absolutely mind blowing - I ran them inside Chrome with a reasonably old graphics card (a three year old GEForce 9600 GT) and the results were incredible - all ran at around 60 frames per second.

A couple of real standouts here:
Floating soap bubbles - fresnel shader demo (rotate view with mouse)
Model of a head - normalmap demo (rotate view with mouse)
Model of Walt Disney - cube mapping and texture demo (rotate view with mouse)

Follow @dodgy_coder

Subscribe to posts via RSS

Modern Cross Platform Development

2012-01-04T02:06:00.000-08:00

Why isn't there a modern technology available for using the same codebase to produce native apps on all of the currently popular platforms - I'm talking iOS (iPhone/iPad/iPod Touch), Android, Windows, Mac and Linux?

That was my original question before I started looking, and since then I've discovered there actually are plenty of new options out there for cross platform development catering for all of the above platforms.

A Brief History of Desktop UI Toolkits

In the 1980's the problem of a cross platform desktop user interface was for the most part solved by the X Window system, known as X11, (1984-) along with one of the first widget toolkits, Motif (1989-), which was built on top of X11. Back then, the dominant platforms used by business were Microsoft Windows and the various flavours of Unix, like Sun Solaris, HP-UX, IBM's AIX etc (later Linux came along and maintained full support for X11 and Motif). Developers who wanted to target multiple platforms such as these had this option of developing one codebase, usually written in C/C++, using X11 and/or Motif. The UI code for the most part could remain the same and the application just needed to be recompiled with the provided X11 libraries.

Since then Motif has pretty much faded into the background and been replaced with newer widget toolkits, still built on top of X11, including Qt (1991-), wxWidgets (1992-) and GTK+ (1998-). These now run on many different platforms and bindings are available for many, many languages, including for their native C or C++.

These three free open source toolkits have been successful for the desktop application case but looking towards mobile and tablet platforms, these toolkits don't currently have the support there to take them into the future.

Qt
Native language: C++
Platforms: iOS (unofficial), Windows, Mac, Linux.

Probably the C++ based Qt widget toolkit is the most well established way of writing desktop cross platform applications. Smartphone support isn't really there yet, but there is an unofficial port to iOS. Linux's KDE desktop environment is written with the Qt toolkit. Many significant apps have been developed with Qt, including Autodesk Maya, VLC media player, Mathematica, Virtual Box and Skype. Qt is now owned by Nokia (although is still open source), and there was a release of a new version recently (Qt 4.8).

GTK+
Native language: C
Platforms: HTML5 (unofficial), Windows, Mac, Linux.

The C-based GTK+ widget toolkit is also a very well established way of writing cross platform desktop apps. Smartphone support isn't there, but there is an unofficial port to HTML5. GTK+ powers Chromium on Linux, the open source forerunner to Google's Chrome browser. The GNOME desktop environment is written with the GTK+ toolkit. The other notable app written with GTK+ is the one that launched the toolkit itself, GIMP, an open source bitmap image editor.

wxWidgets
Native language: C++
Platforms: iOS, Windows, Mac, Linux.

A C++ native mode toolkit that provides a thin abstraction to a platform's native widgets. It was originally developed as a desktop GUI toolkit in the same vein as GTK+ and Qt and support for iOS is still beta. There are a number of notable apps developed with wxWidgets including BitTorrent, Audacity, TortoiseCVS and RapidSVN.

Java and Silverlight

Sun's Java Swing UI Library (1997-) and Microsoft's Silverlight (2007-) were both attempts at "write once, run anywhere" (WORA) technology that provided a rich user interface library. Although technically successful, they failed in their attempt to become the ubiquitous technology that everyone uses to develop UIs. They both attempted to replace the 'native' look of apps developed on a particular platform and instead impose their own look and feel based on graphical primitives, which although being more powerful for developers, users on the whole didn't like.

As Johannes Fahrenkrug discusses here, people actually want to run Windows apps that look like Windows apps, and run MacOS apps that look like MacOS apps. To be fair, the Java Swing UI has actually undergone a lot of advances since it was first released and can now generate apps that do look perfectly native on all desktop platforms. The problem may be however that people's original experience of early Swing apps having a slow and inconsistent UI might have put them off using other Java apps and so contributed to its demise.

In the case of Microsoft's Silverlight and WPF, although not officially dead yet, they are now taking a back seat to newer technologies that were released by Microsoft at the September 2011 Build conference; Windows 8 allow developers to create new style 'Metro' applications by using either web technology (HTML5/JavaScript/CSS) or a traditional language - C#, VB.NET or C/C++ paired up with XAML for describing the UI. There's more detail about this topic on StackOverflow here for those interested.

The Rise of Web Applications and the App Store

The two big changes since the desktop UI toolkit days has been the rise of browser-based web applications and more recently the rise of downloadable apps running on smartphones and tablets such as Apple's iOS (powering the iPod touch, iPhone and iPad devices) along with Google's Android (powering a vast array of different smartphones and tablets from many hardware vendors). Google's recent Chrome OS also deserves a mention for creating an OS around the browser itself. These new class of devices offer a quick method of downloading, paying for and installing apps (often with just one click) and appear in the easy to use 'App Store' on iOS and 'Android Marketplace' on Android.

Microsoft is getting in on the App Store action a little late with its 'Windows Store' for Windows 8, which will be appearing some time in 2012. Its thought by Hal Berenson (a former Distinguished Engineer at Microsoft) that the upcoming Windows Phone 8 OS will be based on the same technology as Windows 8 and hence will unify the development platform for Microsoft's desktop, tablet and smartphone operating systems. This will likely be announced some time in 2012, probably at the same time Microsoft launches the 'Windows Store'. This will mean Windows 8's new 'Metro' style apps on the Windows Store will run on phones, desktops and tablets - something that neither Apple nor Google can lay claim to, just yet. Add in the possibility of an XBox update which adds support for running Metro apps, and you have a huge carrot which will make the unified Windows 8 platform very attractive to both application and game developers.

Choosing Your Platform - Desktop, Web or Mobile Application?

When choosing your platform, if you need full access the underlying file system, or access to hardware devices like serial ports, microphones, webcams, etc, then you may be better off staying with a desktop application. If you need platform independence on top of this then you can go with one of the traditional widget library technologies like wxWidgets, GTK+ or Qt to provide a full cross platform UI solution.

Alternatively, if you don't need so much access to hardware and can remain restricted to the browser, then HTML5 and JavaScript may be the way to go. Browser based web applications have their own set of problems however, including the browser incompatibilities, varying support for the HTML/CSS web standards and the Document Object Model (DOM). In addition, web application development generally makes it harder to create rich UI apps than for desktop or mobile applications. HTML5 will provide massive improvements, but web apps are still running in a browser and so will always remain sandboxed and limited in what they can achieve (no access to file system, serial ports, microphone, webcam, etc.).

The main advantage of the browser app however, and its a massive one, is that its everwhere and supported by every device, so if HTML5 takes off, then it could become the defacto standard for developing apps of any kind, desktop or not. As Jeff Atwood discusses in his seminal blog post All Programming is Web Programming; "the web is the most efficient, most pervasive, most immediate distribution network for software ever created - its almost completely frictionless". As Atwood mentions, more and more users gravitate towards preferring to run their apps inside the browser, on smartphones and on tablets where things 'just work' and there's generally nowhere near as much friction caused by the traditional desktop software setup and update process.

A third option available now is the cross platform mobile development SDK as detailed below (Mono, Appcelerator Titanium, Rhodes, PhoneGap, MoSync, Moai, Corona SDK and JUCE). Smartphone and tablet based applications are taking off, and these SDKs allow you to write once and run on any of these new class of devices, as well as in some cases also being able to run on the traditional desktop. These would seem to offer some of the 'power' of the desktop in terms of native code execution speed and access to hardware functionality, along with the enourmous 'frictionless' benefit that you get when publishing to an app store, where installing an app is literally a one click operation.

Mono
Native language: C#
Platforms: iOS, Android, Windows, Mac, Linux, Browser (e.g. via ASP.NET WebForms or MVC).
Cost: Windows, Mac & Linux: free. Android and iOS: from US$399 each. Free trial available.
License: LGPL/GPL/X11 combination. Commercial license available.
Source: Mono for Windows/Mac/Linux is open source. Mono for iOS/Android is closed source.

The C# based Mono cross platform library allows apps to be built using purely C#. However this is not strictly a cross platform UI library, since each supported platform has its own different set of Mono C# UI libaries, each with their own capabilities. The idea is that you build your software using the MVC (Model View Controller) pattern, so that the Model and Controller components (both written in C#) can be shared across all platforms without any changes. Only the View component needs to be re-written for each platform (again, in C#). Mono targets both mobile and desktop apps, however if you are just looking at developing a desktop application, then Mono probably isn't a big improvement over GTK+ or Qt, after all a well written C++ Library (with a Model and Controller component for example) will enjoy the same capability to be shared across more than one platform if you use Qt or GTK+ and then you wouldn't need a different UI library on each platform to create your view. There are several advantages that Mono has over Qt or GTK+ however - firstly in leveraging a team's existing knowledge (and codebase) of C# which is a popular language. Secondly, being able to target iOS, Android and the browser. Lastly, via Mono's .NET CLR implementation (Common Language Runtime), code is compiled into modules of bytecode, which are binary compatible across all the supported platforms.

There is also an open source library for Mono called MonoGame which lets you port with minimal effort XNA based games (which originally target the .NET platform and run only on Windows Mobile, Windows and XBox) to any of the Mono platforms such as MacOS, Linux, iOS and Android.

Appcelerator Titanium
Native language: JavaScript, Python and Ruby
Platforms: iOS, Android, Windows, Mac, Linux.
Cost: Community Edition is free. Indie Edition (extra APIs, help, support): US$499 per year.
License: Apache Public License v2
Source: Open source

An open-source solution which allows you to use web development languages (HTML, JavaScript, CSS, Python and Ruby) to build mobile and desktop apps. App data can be stored in the cloud or on the device, and apps can take full advantage of hardware, particularly camera and video camera capability. Appcelerator apps go through a complex compilation and optimization process which result in it creating apps that look, feel, and perform just like native apps, because you are using native UI components... so an iPhone app developed with Titanium will actually feel like an iPhone app. They also offer an API repository marketplace to buy and sell code to extend Titanium. One of the downloadable components allows you to develop your app using an MVC (Model View Controller) pattern. There is an active developer community Q&A site for Titanium developers which is similar in usage to StackOverflow.

Rhodes
Native language: Ruby
Platforms: iOS, Android, Windows Mobile, Blackberry.
Cost: Free
License: MIT License
Source: Open source

Rhodes is an open source, Ruby-based framework that allows the development of native apps for a wide range of smartphone devices and operating systems. The framework lets you write your code once and use it to quickly build apps for every major smartphone. Apps can take full advantage of available hardware, including GPS and camera, as well as location data. Rhodes enforces a strict MVC (Model View Controller) pattern on your apps. Views are written in HTML5 while Controllers and Models are written in Ruby.

PhoneGap
Native language: HTML5/JavaScript
Platforms: iOS, Android, Windows Mobile, Blackberry, Symbian, Palm.
Cost: Free
License: Apache License, Version 2.0
Source: Open source

PhoneGap is an open source framework that helps you develop apps for smartphones using web development languages such as JavaScript and HTML5. It also allows for access to hardware features including GPS/location data, accelerometer, camera, sound and more. PhoneGap lets you take a web app, run it in a UIWebView, and through javascript you can access iPhone features such as the camera an accelerometer. This means that it won't necessarily have the look and feel of a native app (like the other cross platform SDKs listed here). The idea is that you develop your web app like you normally do, and then use PhoneGap to 'bridge the gap' to the phone, and give you access to phone's hardware, that traditionally you couldn't access from a web-only app.

MoSync
Native language: C++
Platforms: iOS, Android, Windows Mobile, Blackberry, Symbian.
Cost: Free for Indie developers. From €199 (Euros) per year for Enterprise developers.
License: GPL v2. Commercial license available.
Source: Open source

The MoSync Mobile SDK is a cross-platform mobile application development SDK which allows the development of native apps. Free open-source, it is based on C++ but allows development also in HTML5 and JavaScript. MoSync provides access to a wide range of underlying device functionality, via a C++ SDK or optionally via JavaScript. It requires probably a greater degree of software skill from the developer, but in principle can enable a rich variety of different kinds of application.

Moai
Native language: Lua
Platforms: iOS, Android, Chrome Web Store, Windows, Mac, Linux.
Cost: Free
License: Common Public Attribution License Version 1.0 (CPAL)
Source: Open source

Although intended to be a mobile game development SDK, I think it deserves a mention here due to its unique scripted approach to cross platform development. Games developed with Moai run on iOS and Android as well as the Chrome Web Store, Windows, Mac and Linux. Designed to be programmed with the Lua scripting language, Moai is designed for experienced game developers who wish to use Lua for mobile development. The Moai framework itself is written in C++ and is intended to be called from the Lua scripting language, but can also be called from any other language supported by the host platform. The Lua script is run as interpreted byte code, but since very few Lua instructions are processed among all the input handling, this means that rendering, animation, collision detection, and physics math runs natively. Moai's physics engine for example is the Box2D open source C++ library. Moai is free and open source, however to build apps for iOS devices you'll need the iOS SDK and the Xcode IDE (presumably running on MacOS) and be a member of the Apple iOS developer program (US$99/year). To build apps for Android you'll need the Android SDK, the Eclipse IDE and an Android device.

Corona SDK
Native language: Lua
Platforms: iOS, Android, Kindle Fire, Nook Color.
Cost: US$199/year for Android. US$199/year for iOS. $349/year for all platforms. Free trial available.
License: Commercial, price as above.
Source: Closed source

A similar toolkit to Moai in that its primarily aimed at game developers, and apps are written with the Lua scripting language. Corona SDK was developed by two former Adobe mobile engineers and has been in development since 2007. It features a proprietary OpenGL-ES rendering engine, which allows for full hardware acceleration of graphics, including sprites that animate at full GPU speed. The Corona physics engine is built around Box2D as with Moai. Its free to try Corona and run your apps in a simulator on your desktop computer, but to build an app and publish it to an App Store, you'll need to pay a yearly subscription fee (of either US$199 per year for Android or iOS or the premium subscription of US$349 for all platforms). Your apps are built "in the cloud" on Corona's server. Note that if you are developing for iOS you'll still need to pay the yearly Apple iOS developer program fee which is currently US$99. There seems to be extensive documentation available for Corona and an active developer community forum. There has been a recent push to promote Corona for use not only for developing games but also for general apps and also interactive ebooks.

JUCE
Native language: C++
Platforms: Windows, MacOS, Linux, iOS, Android (Note that Android support is a work in progress).
Cost: Free for GPL license. Commercial license from £399 (UK pounds) per product.
License: GPL. Commercial license available.
Source: Open source

JUCE is an all-encompassing C++ class library for developing cross-platform software. Its designed to contain everything you're likely to need to create most applications, and is particularly well-suited for building highly-customised GUIs, and for handling graphics and sound. JUCE is developed by Raw Material Software and consists of a small team of developers based in London, England, founded by Julian Storer in 1999, who is still the primary developer today. It would probably be best suited to experienced C++ developers. Julian Storer himself is a professional C++ coder with over 15 years of C++ experience; he takes pride in the fact the source for JUCE is literate, coherent, cross-platform and high quality. Adding JUCE to your C++ application is very simple - the easiest way involves simply including juce.h in your files and adding a single .cpp file to your project. No need to compile any libraries or worry about dependencies.

Follow @dodgy_coder

Subscribe to posts via RSS

"Yoda Conditions", "Pokémon Exception Handling" and other programming classics

2011-11-27T02:44:00.000-08:00

Just reading through some of the excellent answers to this question posed on StackOverflow.com by John K

What programming terms have you coined that have taken off in your own circles (i.e. have heard others repeat it)? It might be within your own team, workplace or garnered greater popularity on the Internet.

Reposting some of the best ones below ...

Yoda Conditions

Using if(constant == variable) instead of if(variable == constant), like if(4 == foo).

Because it's like saying "if blue is the sky" or "if tall is the man".

by [zneak]

Originally, yoda conditions might have been introduced to reduce the potential of coding errors in the form if (5 = count) which would be picked up at compile time.

Pokémon Exception Handling

For when you just Gotta Catch 'Em All.

try
{
// do something
}
catch
{
// catch em all
}

by [woot4moo]

Egyptian brackets

The style of brackets where the opening brace goes on the end of the current line ...

if (a == b) {
printf("hello");
}

This style of brackets is used in Kernighan and Ritchie's book The C Programming Language, so it's known by many as K&R style.

by [computronium]

Refer to the Wikipedia entry on Indent style for further reading, including 1TBS (The One True Brace Style), Allman style (also known as ANSI style) and Whitesmiths style.

Different kinds of bug reports

Smug Report - a bug submitted by a user who thinks he knows a lot more about the system's design than he really does. Filled with irrelevant technical details and one or more suggestions (always wrong) about what he thinks is causing the problem and how we should fix it. by [aaronaught]

Drug Report - a report so utterly incomprehensible that whoever submitted it must have been smoking crack. The lesser version is a chug report, where the submitter is thought to have had one too many. by [aaronaught]

Shrug Report - a bug report with no error message or repro steps and only a vague description of the problem. Usually contains the phrase "doesn't work." by [aaronaught]

Fermat's Last Post - a post to a bug tracker/email list/forum in which the author claims to have found a simple fix or workaround for a bug, but never says what it is and never shows up again to explain it (even after others have been puzzling over the bug for years). by [alan moore]

Floater - the case in a bug tracking system where a bug report remains "floating" at the top of the queue, but never gets assigned to a developer, maybe because there is a workaround in place.

Refuctoring

The process of taking a well-designed piece of code and, through a series of small, reversible changes, making it completely unmaintainable by anyone except yourself.

by [dan-vinton]

A play on the term Refactoring which was popularized by Martin Fowler in his book of the same name.

Stringly Typed

An implementation that needlessly relies on strings when programmer and refactor friendly options are available. For example, method parameters that take strings when other more appropriate types should be used. Excessively stringly typed code is usually a pain to understand and detonates at runtime with errors that the compiler would normally find.

by [mark-simpson]

A play on the term Strongly Typed, code which restricts the mixing of different or incompatible data types, allowing errors be picked up at compile time.

Different kinds of bugs

Heisenbug - a bug that disappears or alters its characteristics when an attempt is made to study it. by [jacob]

Higgs-Bugson - a hypothetical bug predicted to exist based on a small number of possibly related event log entries and vague anecdotal reports from users, but it is difficult (if not impossible) to reproduce on a dev machine because you don't really know if it's there, and if it is there what is causing it. To find these you will need to invest in a Large Hadron debugger. by [runrunraygun]

Hindenbug - a catastrophic data destroying bug - "Oh the humanity!". by [mike-robinson]

Counterbug - a bug you present when presented with a bug caused by the person presenting the bug. by [mike-robinson]

Bloombug - a bug that accidentally generates money. by [mike-robinson]

Schrödinbug - refers to a function/feature that appears to fluctuate between buggy and correct (like Schrödinger's cat, fluctuating between alive and dead), until somebody looks at the source code (opens the box), at which point it becomes permanently bugged. by [aaronaught]

Loch Ness Monster Bug - a bug which cannot be reproduced or has only been sighted by one person. by [russau] (Also, Bugfoot is a great alternative).

UFO bug - a bug presented by customers, who, even when they're shown that it doesn't exist, will repeatedly report it again and again, believing it is real. by [Tuqui]

Mandelbug - a bug whose causes are so complex that its behavior appears chaotic or even non-deterministic. Named after fractal innovator Benoît Mandelbrot. by [coehlo]

Brown-paper-bag bug - a bug in a public software release that is so embarrassing that the author notionally wears a brown paper bag over his head for a while. by [ESR via Jargon File]

Sorcerer's apprentice mode bug - a bug in a protocol where, under some circumstances, the receipt of a message causes multiple messages to be sent, each of which, when received, triggers the same bug. by [ESR via Jargon File]

Mad girlfriend bug - a bug whose immediate effect remains hidden - the app outwardly seems to function normally and tells you that everything is fine. by [jeduan-cornejo]

Excalibur bug - when all of the developers within a company have tried to fix a particular bug but none ~~are worthy~~ have succeeded, so far.

Load-bearing printf bug - when a line of debug output is required for the code to work - the code does not function if you remove it. by [Ken]

Doctype Decoration

When web designers add a doctype declaration but don't bother to write valid markup.

by [zurahn]

New software features

A Unicorny - A feature that's so early in the planning stages that it might as well be imaginary.

A Barack Obama - A feature we'd really like to add to a project but will probably never get approval for.

by [jordan]

Different types of Code

Spaghetti Code - Code that's got one too many GOTOs, exceptions, threads, or other "unstructured" branching constructs. Program flow tends to look like a bowl of spaghetti - twisted and tangled.

Spaghetti with Meatballs Code - An attempt at object-oriented code, but where the end result still remains dependent on some messy procedural (spaghetti) code.

Baklava Code - Code with too many layers. Also known as Lasagne Code. by [john-d-cook]

Ravioli Code - Object-oriented code consisting of a number of small and loosely-coupled software components.

Sausage Code - Once you've examined the code in detail ("how it's made") you'll never want to use it again.

Jenga Code - The whole thing collapses when you alter a block of code. Also known as Crispy Noodle Code.

Hydra Code - Code that cannot be fixed. One fix causes two new bugs. It should be rewritten. by [nick-dandoulakis]

Scar Tissue Code - Any code that is commented out but still included in the current and/or checked-in version. by [kelly-french]

Hooker Code - Code that is problematic and causes application instability (i.e. the application "goes down" often).

Pig's Lipstick Code - Code which has large amount of legacy and/or spaghetti code hidden via wrapper objects, so while it appears to new developers to be well designed, elegant, object-oriented code, when they start working on it in depth they realize just how ugly it really is.

Mortgage Code - Code purposely so terrible and complicated that only you can maintain it, forcing your employer to keep you, thus providing job security (so you can pay your mortgage).

Ghetto Code - A particularly inelegant and obviously suboptimal section of code that still meets the original requirements.

Swiss Army Code - Application code that is suffering from feature creep - it does a lot of things, but it does nothing well.

NP Tricky Code - An algorithm whose complexity is too hard for a mere mortal to figure out.

NP Hilarious Code - An algorithm whose complexity is "a joke", literally (as in BogoSort) or metaphorically.

Cut-and-waste Code - When someone cut and pasted code they found online (usually from a blog) into a production product; the result is usually a lot of wasted time trying to track down an obscure bug from code that undoubtedly made sense in the original context but not in our app. Also known as BDD - Blog Driven Development.

Neighborhood Bike Code - A module or a piece of code that every programmer at the company has touched.

Eraser Stains Code - Code that has been written, then re-factored multiple times leaving swaths of legacy code and design. Like erasing a sheet of paper so many times, the pencil marks are no longer the problem - the large greasy stain is.

Objectfuscated Code - Object-oriented code which has been abstracted to so many levels that no-one can understand it anymore.

Barnacle Code - Any piece of code (usually a static method) which has been appended to a class where it doesn't really belong, due to a lack of anywhere else to logically put it.

Autopilot Code - Code that was written by a programmer who was on 'auto-pilot' or wasn't really thinking about what they were doing.

Fear Driven Development

When project management adds more pressure (fires someone or something).

by [arnis-l]

Protoduction

A prototype that ends up in production.

by [chris-pebble]

Ninja Comments

Also known as invisible comments, secret comments, or no comments.

by [schar]

Smurf Naming Convention

When almost every class has the same prefix.

by [sal]

Thanks again to the above answerers from the original StackOverflow question:
[zneak][woot4moo][computronium][aaronaught][dan-vinton][mark-simpson][jacob][runrunraygun][mike-robinson][russau][zurahn][jordan][john-d-cook][arnis-l] [nick-dandoulakis][chris-pebble][schar][sal][kelly-french][jeduan-cornejo]

And to the original poster of the question: John K

If you have any good suggestions, please add some comments below and I'll update the post.

Follow @dodgy_coder

Subscribe to posts via RSS

Why do Programmers get paid less than PMs and BAs

2011-11-12T19:37:00.000-08:00

I came across this interesting question over at programmers.stackexchange.com which asked: Given that programming is generally more difficult, why do business analysts and project managers get higher salary than programmers? What is it that makes their job a high paying job when even at most times programmers are the ones that go home late?

There were a few great answers, but only one stands out as the truth, and that is the one which is coming from an economic rationalist's viewpoint ...

People get paid less than their employers' "reservation price" (the most they would ever pay), and more than their "marginal product" (the least they would ever work for). Your actual pay on that spectrum is based on your bargaining power, relative to your employer.

Say your services to your company are worth $1000 a day. Under the gun, he would pay you that if he had no other choice. Say you would work for $100 a day if you had no other options. That's your range.

Say you're new and independent and unknown, and your boss is Google. Google has more bargaining power, because they can just wait and hire someone else, and lots of people want to work for them. You have less power, because you have to pay rent, so you'll get closer to $100 than $1000 a day.

Say you're the last COBOL programmer on Earth, and your boss's mainframe runs COBOL. Then you have more bargaining power, and will get closer to $1000 a day.

SO, either PM/BAs are worth more to the company, OR they have higher bargaining power. I don't think it's the first option, so must be the latter. Good people skills are rare. It's also hard to outsource them, as they have to meet clients. Their relative scarcity gives them more bargaining power, and thus higher pay.

This is a great answer by NRM and although slightly boring compared to the other answers which touch on the political side of things, it definitely has the ring of truth to it. As he mentions, good people skills are rare, and provide a huge benefit in any job, especially in the field of software development.

Follow @dodgy_coder

Subscribe to posts via RSS

StackOverflow's Programming Language Bias

2011-11-05T23:15:00.000-07:00

Using StackOverflow.com, I've always had the impression that its biased towards C#, and .NET in general. That might have been because on the SO tags page C# is number one, with the most questions tagged, or maybe because much of the site itself is built with C# and MVC.NET. So I thought it would be interesting to compare the rankings of the tag popularity on StackOverflow with a leading language popularity index (the TIOBE index).

Method
The "Stack Overflow Representation" represents a ratio of the SO tag count (as a percent of total questions) divided by the TIOBE language index percent. Where a representation of 100% means that the SO tag count is aligned exactly with the TIOBE language index. An "over-representation", greater than 100%, might mean there's a greater number of questions on SO than we'd expect. An "under-representation", lower than 100%, might mean there's not as many questions on SO than we'd expect.

Results
Suprisingly, JavaScript came out to be the most "over-represented" language on SO, by quite a long way at 294%. Could this also be because programming JavaScript is generally quite difficult and will result in people seeking help more often? Following this was C# (which I had expected to be number 1), at 153%. After this, PHP, Ruby and Python were basically fairly balanced at around 100%. The most "under-represented" major language would definitely be C at 11%. Three other major languages which seemed to be a bit under-represented, below 50%, were C++, Java and Objective-C.

Data Sources:

Stack Overflow data taken from: StackOverflow.com/tags - data snapshot taken at 2:50am UTC, 6-Nov-2011. Total tag count was 2,254,639.
TIOBE data taken from: latest available (October 2011) at www.tiobe.com/index.php/content/paperinfo/tpci/index.html

Follow @dodgy_coder

Subscribe to posts via RSS

How to Become a Better Programmer

2011-10-01T00:58:00.000-07:00

Here then, in no particular order, are twelve positive things that you can do, at any point in your career, to improve your programming skills. These will also have the added benefit of injecting additional passion into your programming ...

1. Never Stop Learning and Reading

Consciously try to improve yourself and the quality of code you write. Think about what you're doing, why you're doing it, and how you can do it better. Look back at what you've done on previous projects, and how you might do it differently now, or where you could still improve on it.

Read books, not just websites.
Read for self-improvement, not just for the latest project.
Read about improving your trade, not just about the latest technology.

2. Work With People Smarter Than Yourself

Working with smarter and/or more experienced developers will teach you a great deal. You can learn a lot by reading their code, watching their practices and listening to their opinions and stories. Also, you should always listen to what others have to say, regardless of whether they're junior, intermediate, senior or guru-level developers.

3. Become a Polymath (or 'Jack-of-all-Trades')

Decide to be a 'Jack-of-all-Trades' - also known as a Polymath, Polyglot, Generalist, Renaissance Man or Multi-specialist. This will allow you to avoid becoming 'pigeon-holed' into one specialty, which can stagnate your programming skills, as well as hurt your future employment prospects if your 'specialty' suddenly becomes yesterday's technology. Something to keep in mind is 'the half-life of knowledge' law of technology; it tracks with Moore's Law: half of everything you know will be obsolete in 18-24 months. An expert who chooses the wrong discipline can easily be undermined by the press of technology; a generalist only has to add some more skills and remember the lessons of the past in applying those skills.

4. Read and Document Other People's Code

Read and attempt to understand code written by others. Plus, write documentation for code written by other people. Writing code is significantly easier than reading someone else's code and figuring out what it does. You can see how they do something - maybe they do it in a better way than you. And sometimes this can be one of the best ways to learn how not to do something.

5. Get Programming Experience on a Real Project

There is nothing like getting in and coding, especially under pressure - work on a real project, with real fickle customers, with real, ever-changing requirements and with real engineering problems.

6. Teach Others About Programming

Take a part-time job tutoring computer science students at a local university or college. This will force you to understand something at a completely different level, since you have to explain it to someone else. As Douglas Adams wrote, "the best way is to try and explain it to someone else. That forces you to sort it out in your mind. And the more slow and dim-witted your pupil, the more you have to break things down into more and more simple ideas."

7. Learn One New Programming Language Every Year

One year gives you enough time to get past the basics - it pushes you towards understanding what's beneficial in that language, and to be able to program in a style native to that language. Each language will change the way you think about programming, and how they do things, to compare that to what you already know.

8. Complete One New Pet Project Every Year

Start a "pet" project and follow it to completion and delivery. A good pet project will push your boundaries and keep you interested; it may be something you have experience with and enjoy plus something you don't know about. E.g. if you know something about game development but don't know Ruby, a good pet project may be to develop a game written in Ruby.

9. Learn Assembly Language

Learning a low level language like assembly gives you insight into the way computers 'think' without any high-level abstractions; the elegance at this level is surprising. There are no wasted motions, no disposing of data, and bugs can have a far greater impact. Developing at this level will teach you efficiency and hone your critical thinking and logic skills.

10. See Your Application From the End User's Perspective

Step out of the echo chamber and improve your understanding of the end-user aspect of software development. Interact with the end-user to see, through their eyes, how they use your software. End users are typically not technical, and they often see software as a magical piece of work, while you see software as a logical set of steps. The two worlds are completely different - what seems easy and logical to you may seem cryptic and inscrutable to others. You can also find out what kind of 'abhorrent' information they throw at your application. This will help you generate good test cases and stop assuming your program will always be fed the correct set of data.

11. Start a Physical Exercise Program

You work a whole lot better when you're in good physical shape - problems become easier and less overwhelming, wasting time is much less of a temptation, you can think clearer, and working through things step by step doesn't seem an arduous task. An old one but here it is: "Sound body, sound mind.". Regularly devote a bit of time to physical exercise - riding a bike, walking, running, swimming, whatever suits you best.

12. Learn Touch Typing

Typing for a programmer is essential. Everyone has had a coworker who typed using just two fingers and had to look at the keyboard for everything before pecking out their code one letter at a time! Learning to touch type is a quick and effective way to give your productivity a boost as a programmer.

Follow @dodgy_coder

Subscribe to posts via RSS