Spillikins №174. What Benchmarks Mean

Last week I focused on testing Galaxy S3 and spent little time on other phones. A few days ago, I posted a video comparison of HTC One X with Galaxy S3 where I highlighted only a few features. I apologize for my mistake: One X actually supports Swype out-of-box. However, this mistake does not really matter, as the point of the video was to highlight the strong and the weak sides of each device. I did not expect such a hot dispute between HTC and Samsung fans over this video. It seems that even though there are a lot more Samsung supporters HTC fans are way more critical. I am glad my video spurred so much discussion and got over 20,000 views already.

I hope that Galaxy S2 owners catch up and begin defending their phones against S3. By the way, the full Galaxy S3 review is coming very soon meanwhile you can check this video.

The Clash of Benchmarks

There are two kinds of people: those who believe in superstitions and those who don't. For example, a black cat crossing your path is an ill omen in some cultures. To some extent manufacturers today are trying to use superstitions to create loyal customers. It is now important right now who started this game but it is another game of stereotypes. In the last Spillikins I talked about camera megapixels and real art, horse power and driving skills. Unfortunately, some people are sensitive even to those platitudes. But it is not surprising since we go hand in hand with myths and stereotypes. I merely tried to shed tome light on a myth but some people felt like I dared to invade sacred grounds. Today I want to continue talking about this fruitful topic but from another angle: what do smartphone benchmarks mean?
Many young men bench their smartphones the moment they get them out of the box. Long before smartphone this game had been popular with PC users and I too used to bench my computers. Windows has even got a performance index of its own. It also uses a virtual performance scale that means nothing. Today, in the age of smartphones there is a myriad of benchmarks available but what people who use them really want to know is whether their phone is better than the phones of their friends.
To get things straight let’s see what benchmarks do. Benchmarks use a series of typical tasks to assess the performance of the CPU, memory and other components. Depending on the skills and the talent of the author such tests can be a pretty accurate approximation or have nothing to do with reality. Number one problem of any benchmark is that they are only applications to an OS. The operating system allocates certain resources, a ‘playground’ in which all apps operate. So any app’s performance is restricted by the OS. as a result a benchmark can only assess the performance of an app it itself being one. Of course, we can then correlate the results of the benchmark performance and the hardware general performance but it is a very hard job. Besides, smartphone manufacturers add their own UI overlays and tweak the original OS, which also hampers a benchmark’s accuracy and validity.
Another factor is that most benchmarks don't take into account software optimization. Try launching a game on Tegra3 that was designed specifically for Tegra3 and then try playing it on another platform – the result will be lamentable. However, you will probably be able to find a low-res version of a game for simple devices. I want to say that in this case the number of benchmark points are completely irrelevant for the user – it only matters for him whether he will able to play a game. The benchmark logic would suggest that Tegra3 has the best performance since it can launch some games other devices can't but it's not true – it is a matter of software optimization. NVIDIA puts a lot of effort into optimizing games for their chipsets.
Tegra3 on HTC One X does worse than Galaxy S3 in the Quadrant Standard benchmark test but does it mean it is a slower platform? They both have the same amount of RAM but S3 has actually a lower CPU clock. I guess my conclusion will cause tons of angry comments again: these results mean nothing at all. The only valid corollary of a benchmark test is how well a device can run the benchmark test and that is it. A benchmark cannot tell you how what its figures mean in regard to user experience: does the device start faster? Or may be it takes photos faster? Benchmarks don't know that. Modern smartphones are extremely complex sets of very complicated components and each of them has a huge effect on the performance in general and benchmarks cannot tell you what are the strong and weak sides of your device.
Let's leave synthetic tests aside for a minute. Imagine that your company is a major manufacturer and the market leader. There are many rivals who offer very decent competing products. To prove superiority of your products you create a benchmark that assesses performance of your devices and the rivals' ones but it is tailored to your products and disregards peculiarities of the competing products. Can such a test be a valid comparison? I think it would be utterly pointless even if the goal was an fair test.
Before looking at benchmark figures you should always remember that they are relative. Benchmark points stand for performance of one designated task on on operating system. However, benchmarks can be used to assess the performance of a device before and after firmware update or with power saver mode on or off. But benchmarks can never tell you which device is a better choice. There are no benchmarks that could validly assess the performance of iPhone, iPad, Android, WP7 (7.5) and other OS altogether. Synthetic tests can only be accurate within one platform. The only theoretical exception is browser comparison but in reality a browser performance depends on a multitude of factors like the screen resolution and optimization level.
So, are benchmarks useless? On the one hand, yes as they cannot tell you which device is better. However, they can give some objective data that can be used in a comparison. HTC One X with Qualcomm S4 does brilliant in synthetic tests and not so good in other tests – which one should we believe? I have come across a very interesting chipset comparison.
The first picture shows Tegra3 at max details, the second one S4 Qualcomm with few details and the performance is similar. So, which one is better?
The verdict should be based on actual user experience and not on some benchmark points. Besides, games are not everything there are many people who care about the performance of office apps but, unfortunately, no one cares to bench it. I hope this little note will help you to better understand benchmark figures and make the right choice the next time you choose a phone. Remember that benchmarks do not assess the general performance but only a handful of tasks.

Android OS Statistics

Android developer blog posted fresh statistics on the distribution of the OS versions which is particularly interesting with the growing sales of Ice cream Sandwich. The data shown as of late march:

As you understand, as of June the distribution has not changed a lot: the share of Android tablets is growing (Android 3.x) as well as the share of the latest Android version ICS. Surprisingly, the share of Android 2.3.x devices is also growing. The reason for that is the wave of very affordable 2.3 Android phones. In this case the OS version is irrelevant – the main factor is that more people can afford an Android.
These data includes all OS copies: OEM and custom builds. It is not important for Google how exactly the user acquired an Android and whether it is an official build they care that people are using their OS. Many overestimate the number of custom builds and I think that there is hardly 1% of the total Android users who use custom firmware. However, it is now very easy to get your hands on one of those on dedicated websites.
I also think that the issue of Android fragmentation is real but its significance is heavily exaggerated. I think that Android sales prove that fragmentation does not bother users or the manufacturers. Anyway, the factor developers most care about is the number of users.

Google's Antitrust Allegations against Nokia and Microsoft

The patent war is getting a new angle and Google seems ready to sue you just about over anything. Last week Google accused Nokia and Microsoft of selling a number of patents to Mosaid, a company known as a patent troll i.e. this company profits from acquired patents and lawsuits over their violation. In September 2011 Nokia sold about 2000 patents to Mosaid. So court ordered payments over those patents will be shared between Mosaid, Nokia and Microsoft. Google thinks this is not fair and filed a lawsuit. However, Google has already presented this case three times in the US courts and lost every time. In every case the court did not see any violation of the free market principles. This did not stop Google and they filed a similar suit but this time for a EU court. Frankly, I cannot fathom how Google can possibly benefit from this but it may be a part of another PR campaign. I suppose the European court is going to refuse any actions all the same just as the US courts did. Anyway, here is an article on this issue:

• Google's EU antitrust allegations against Microsoft and Nokia were dismissed by the ITC

The Art of iOS Icons

Presumably, I saw the links to the original article on Twitter and it was re-posted many times elsewhere. Clearly, the article impressed the readers, though it is only dedicated to the creation of icons for iOS. The success of the article lies in the fact it has many icons and not many words. Several icons can be seen here, but if you want more read the original article.

By Ramotion

By Roman Jusdado

By Konstantin Datz

By Konstantin Datz

By Konstantin Datz

By Aditya Nugraha Putra

By Saturized

By Konstantin Datz

By Dash

By Ryan Ford

By Roman Jusdado

By Artem Borysenko

By Ryan Ford

Nokia: Free Fall Continues

Starting from June 5 the majority of Nokia stores in Russia will be re-branded as Samsung outlets and will sell Samsung Galaxy S3 instead. Ironically, even the flagship Nokia store in Moscow belongs to the rival now. A huge interactive screen has been empty for a month and soon it will bear the name of the main competitor. Things are changing fast these days. Our readers were the first to learn the news.
The next nail in the coffin, which will further exacerbate the situation for Nokia is the decision of other partners not to buy its products and it is a new twist in the tale. Many branded stores will close down as a result of weak sales and lack of support on behalf of Nokia. Instead new outlets selling Samsung or Apple handsets will open. To confirm the trend look at the photo provided by one of our readers.

Source: @alexchernishov

Source: @sdemidov

Sadly, but it is the tip of the iceberg, because we do not get any information directly from the Finnish manufacturer. Judging by those bits we do have the downfall of the company cannot be controlled from within and Nokia pays no attention to the prestige or brand value. Employees of the company are looking for new jobs, while Nokia itself attempts to save on everything it can. For example it sponsored the WRC rally competition, but terminated the agreement at the end of May. The last stage Powered by Nokia was held in Greece. The amount of the deal was 2.5 million British pounds over a period of several years and the company clearly plans to save here. I think it is logical, because Nokia can't advertise anything as it has no products people want to buy.
To assess the magnitude of failure we need concrete evidence, which we can get in second quarter figures to be unveiled in July. I think everybody will be surprised. Not only retailers, but even carriers decide not to promote Nokia products. The speed of destruction is startling. The one piece of data, which shocked me was supplied by GfK Rus. In April 2012 Nokia retail sales in Russia plunged to 20% from almost 35% a month ago. Such a dramatic decline is likely to be repeated on other markets as well. In quarter results we will see absolute declines everywhere apart from the US, but the share of Nokia there is minimal anyway. It is more than a simple catastrophe.
The main problem of Nokia is the absence of products it can sell. There is no single bright model to build upon, while the pricing seems to be questionable to say the least. Nokia Lumia 610 was recently launched in Russia at the price of 295$, though this entry level Windows Phone model with Tango inside has a limited amount of memory and can't run Skype and several other apps. During the first week slightly more than 100 handsets were sold around the country and the price went down to 277$, which is still not little for this type of a smartphone.
Don't forget that many Nokia models in 2012 experience the price drop as a result of low sales. Nokia Lumia 710, which is head and shoulder above 610 costs on average 313$ and can be had for 283$ if you shop online. How can you sell a junior model more expensively then? The answer is blowing in the wind – responsible managers are not professional enough and others cannot be hired under the circumstances, so we will see the continuation of Nokia suffering. The market knows about it and the share prices fall further and further. The market capitalization already reached an embarrassing level of 10$ billion. Even if you sell the corporate property not much more will be added to this extremely low evaluation. Investors have already made their mind and see no future in Nokia. In mid-June the once powerful giant will post record-breaking losses and plummeting sales. The agony will not be long.
P.S. The representative office of Nokia did not even react to the fact that Samsung attached two large boards with their logo. As rumors have it due to redundancies the office will be moved soon to the outskirts, where the rent is cheaper.

Blackberry: One More Lame Duck

I always wondered how similar events unfold at the same time. One Nokia employee said in the private conversation that “the company still has agreements with carriers, which will support the company in Europe for some time, while Blackberry lost even them”. It is not a secret that RIM is fighting for survival and stakes everything on new Blackberry smartphones featuring BB10 to be launched at the end of the year. There is a risk we will not see them on the market though or only some countries will be blessed by their presence. RIM clearly lost their market share and is only thinking what is the best way to surrender.
One of the last-ditch attempts to fight off the imminent demise was the use of external consultants from JPMorgan Chase and Royal Bank of Canada. Both companies have no expertise in the telecommunications sector, but are often involved in the management of mergers and acquisitions. In fact RIM hinted they can sell out to the appropriate suitor. The only issue is that no one is lining up to snap up an ailing Canadian manufacturer. It is difficult to evaluate the price of patents it owns and they are not easy to use anyway. They are either associated with the mobile mail based on the RIM infrastructure not exploited by other companies or deal with QWERTY solutions, which already had their day. Strong points of RIM are not popular nowadays, which will likely to cause the downfall of the brand.
The next step typical of failing corporations is the staff cuts aimed at additional savings. The company has not mentioned the number of employees to lose their jobs, but Reuters expects a whopping redundancies ranging from 6,500 to 16,500. These drastic measures will not help the manufacturer, which leads us to the next point.
We already saw hundreds of millions written off because of Playbook, which flopped. Now warehouse inventory is piling up as the demand for RIM products is decreasing. According to the latest estimates the growth of the warehouse inventory is somewhere between USD 500 million and USD 1 billion. Blackberry smartphones are being replaced by Apple iPhone and Android solutions. This trend is nowhere clearer than in the US. Even corporate networks often use rival gadgets at the expense of Blackberry. The only question is how soon the demand will hit a rock-bottom level. The last straw will be the unveiling of BB10 devices, which are unlikely to be popular. It's a pity, but RIM reaches the end of its road. Frankly speaking the modern market works this way.
Additional reading:

Blackberry World 2012: The Fall of RIM

P.S. Have a nice week and we will try to spoil you with numerous articles, which will not leave you indifferent!