Typically, the amount of defects being reported are expected to skyrocket at the start of the project then decrease as all the existing defects are discovered and reported.  The decrease in defects over time is a good indication of a product that is being thoroughly tested and becoming more stable.  However, this is only true if the reported defects are being resolved.  While the discovery of new defects may decrease over the lifetime of a project it is a dangerous misconception to perceive that because fewer defects are being reported that the product is bug free.  Therefore, taking into account the number of open and unresolved defects in the bug database, the number of open defects must also steadily decrease over time.

With each build deployment that contains fixes, there also contains the potential for more defects to be found.  Immediately after each new build there may be a temporary increase in the number of defects reported.  This is frequently been the case in instances where ‘Critical’ or ‘Blocker’ defects are resolved, since often untested areas within the product become available almost always contain undiscovered defects.  Even with the most minor of defect resolutions, it is always important to retest any area of the product that had been touched by code changes.

Towards the conclusion of development, as the frequency of bug-fixes increases in shorter intervals, it is not uncommon to see a drastic increase in the number of new defects being reported as a result. Consequently, it is not uncommon for rare, often ‘Critical’, defects to emerge shortly before the end of a project.  This is why it is important to test the final bug-fixes in pre-release products despite being confident that the fixes ‘just work’.  Maintaining a contestant QA presence over the lifeline of the product is key to a successful release free of surprise defects.

Businesses are probably aware of the need to build a professional website that is both easy to use and aesthetically pleasing, but did you know that your business also needs a mobile website? Traditional websites are often completely inaccessible from mobile devices including smartphones and tablets.

The Importance of Mobile Websites

Nearly half of all adults living in the U.S. own a smartphone. Many of these smartphone users spend time surfing the Internet from their phones, and some people have even greatly decreased the time that they spend looking at websites on a computer in favor of mobile Internet activity. Even those who do not use smartphones could own tablets that feature mobile computing.

What does this mean for businesses? To put it simply, the increase in smartphone and tablet use means that businesses that opt to design a website based solely on its appearance on a computer are likely to lose potential customers. A company that fails to build a mobile website or mobile app loses out on the business of customers who try to view their website from a mobile device.

Testing Mobile Websites Prior to Launch

Building and launching a mobile website is not enough. Because there are the different smartphone platforms – mainly iPhone and Android; and soon also maybe Windows Phone 8 – companies must test their mobile websites and apps on all platforms they want to support to ensure that their potential customers will be able to access their information and have a good user experience.

Mobile Apps to Assist in Mobile Website Development and Testing

There are a few mobile apps available that help companies test their mobile websites for compatibility with mobile platforms, as well as companies offering mobile emulators as a SAAS model. However, if one truly wants to rule out false positives or negatives during device compatibility testing on actually devices with a variety of service providers will be necessary. Unfortunately purchasing and maintaining an ever growing inventory of mobile devices will be quite pricey and for most companies cost prohibitive.  This is why iBeta’s On-Demand mobile testing service is such a cost effective solution.

But no matter what testing solution one chooses, it is important to keep in mind that when launching a mobile website or mobile app it is important to gain and maintain a loyal customer base. Mobile device software testing is a necessary step that must be made prior to the introduction to the market.  The old mind-set that one can always release bug fixed later during updates, it has proven that the initial customer experience will lay the foundation of how many units can be sold and how many customs will return to the site.

For initial training, all testers have access to a rich variety of immersive QA education documentation & resources. These encompass a wide range of topics not limited to testing procedures, reporting defects, terminology, test cases, and regression. Training is further reinforced with the use of an in-house generated defect-simulator called the iBeta-ulator. To the benefit of both experienced and novice testers, the simulator exposes the testers to the realistic application of standardized testing practices; performing test cases, locating defects, bug writing and reporting the defects into a bug database.

QA education is never limited to just new testers. Often testers also gain project specific experience through the practice of “shadow-testing” (testing in the background alongside ongoing projects.) This practice is especially valuable on projects which frequently expect an increase in tester demand, as it enables more testers with knowledge of the product to be quickly and easily added to a team without diminishing productivity.

Testers are also instructed in maintaining & setting up clean testing environments on a variety of platforms including the latest in Mac OSX, Windows, web browsers, Android & iOS mobile devices, etc. QA Education and training is ongoing, for which iBeta maintains continuous research and knowledge expansion in the areas of OS, Mobile Devices, and mainstream testing methodologies including ‘Agile’. By expanding upon the skill sets of all active testers, this training dynamic allows for maximum flexibility in testing practices, and nearly unlimited adaptability to any project or client needs.

Caveats: The following analysis is based entirely on a brute-force attack and the results cannot be extrapolated to hackers that are using dictionary attacks or have other knowledge that would help them figure out your user’s passwords.

One of these has to do with the “Login page password-guessing attack” that is in the module, Html_Authentication_Audit.script. This script is truly a brute-force attack against a login page, and does not assume that the hacker knows the username of an authentic user of the website. Therefore, even if you lock out a username that is in your database of acceptable users, this particular script has most likely chosen a username or usernames that are not in your database.

Lockout known usernames after failed attempts

First, consider the case that your website locks out known user accounts after some number of failed password attempts. That is, the username entered belongs to a username in your database. After some number of failed logins, your website locks that user out.

Let us assume the following

• Your login page is using TLS or an acceptable variant, so that, for example, if your users are logging in from an unprotected wireless internet café, your login page request is not broadcasting their username and password to the entire café.
• After some number of failed password attempts, you are locking out usernames that you know, at least for some time.

Let there be a hacker! I hereby name my hacker, Hackster. I like to give Hackster perhaps more credit than Hackster is due. The first thing Hackster is going to do is attack the ‘forgot password’ page and see if there is a different response for ‘I cannot find that email address in my list of accounts’ and ‘Hello dude. We sent you a new password!’ If Hackster gets the latter, then he no longer needs to brute force the username, and instead he is going to brute force the ‘forgot password’ page until he has a list of usernames. Suppose Hackster tried that and found out the ‘forgot password’ page was perfect.

Hackster is going to try usernames and passwords both chosen randomly hoping that the combination will hit before the known username lockout hits. Alternatively, Hackster is going to try the same random username with random passwords. What are the odds Hackster succeeds?

Suppose your password policy is weak. You allow 8-character passwords and you do not care what they look like. Hackster has a 50:50 chance of guessing the password after trying 26⁷ passwords. Hackster is no fool and he can easily afford to rent 1000 zombies to attack your website. Supposing it takes 1/5 second to get a response, Hackster will have tried all of those 8,000 million combinations in 18.6 days. Hackster won in less than a month.

Therefore, if the username has an account, it might make sense to lockout that account after, say, 10 bad password attempts. That means that on average across his 1000 zombies, Hackster is going to try a new username every couple of seconds. Hackster may not know during the start of his attack that a lockout will occur, but eventually he will discover that some usernames result in a lockout after 10 bad passwords, while others do not. Hackster will therefore conclude that the usernames that are locked out belong to real users. After that, if the username does not belong to an account, Hackster is also going to stop trying passwords after 10 or 11 bad password attempts, or Hackster is dumber than I thought and is going to waste bandwidth on a futile cause. In the latter case, I would let dumb Hackster do that. Smart Hackster is rolling over to a new username, on the average, every 2 seconds, whether he hit a real username or not. Hackster is hoping for the payoff that one of his combinations will hit. Hackster is trying username and password pairs, so really his chance of guessing is for that combination which is 26¹⁵.  That means we multiply his 18.6 days by 208 thousand-million to arrive at 3.9 million-million days. Smart Hackster has probably done the math and given up.

Hackster has probably considered whether to hit the site with the 11^th password attempt or not. If he does, then he knows whether the account was real or not, but then on the average he has also generated a help-desk call from the real user every 18-36 days. In the latter case, Hackster has a list of real usernames, and can go back after awhile and try another 10 passwords. (Note that there is a clue to your Incident Response Team there. If the user does not call in soon after being locked out, then there is likely an incident in progress or Hackster is also employing social-hacking techniques—those being out of scope of “brute-force”).

Dumb Hackster, however, is generating a help-desk call every 18 days because dumb Hackster forgot to do the math.

Because of the lockout policy, only on real accounts, the website has a reasonable chance of withstanding a brute force attack even with a rather poor password policy. We assumed that the round-trip time was 1/5 second. The calculated days are linear with that time so if the round-trip time is even another 1/5 or 1/10 of that time then the brute-force time does not get a lot better. It does, however, mean that the help desk call frequency will increase by roughly the same factor. Therefore, some attacks could lock out a user every few days.

The previous analysis assumed usernames were relatively simple. Usernames are often case-insensitive, but applying policies to username lengths is not a common practice. Therefore, the username factor of 26⁸ might be as small as 26⁴ or maybe even 26³. If Hackster can target the 3-6 range length of usernames then Hackster could be generating 26⁴ times as many lockouts as described above. Instead of one every few days or few weeks, Hackster is generating 25,000 a day! It probably depends on how many short usernames there are. Hackster might run out of all of them in a few days or weeks, but your help desk is swamped during that time!

If your website has a policy against short usernames or requires full email addresses that are unique as username logins, then a lockout policy on an existing username is quite successful at thwarting a brute-force attack even if the login page simply ignores non-existing username attempts. However, as username-complexity decreases, the number of locked out accounts will increase during an attack, and that may be both a concern as well as a flag of an ongoing attack.

If your website is using a lockout-mechanism to mitigate brute-force attacks of the password of known users, then unless the website also locks out usernames that are not in your database in exactly the same fashion then your website is leaking information about the usernames to the hacker.

Mitigation without lock-out

Suppose that you have no lockout on accounts. Is there a way to deal with brute-force attacks without a lockout?

As described above, depending on the turnaround to your site, a poor password policy can result in Hackster breaking in as little as 18 days if Hackster already knows the username of the account.

First, we consider hardening the password policy. Suppose you have a password policy of 12 characters, one of them has to be a number, and of mixed case. Hackster has a 50:50 chance of getting in within a window of 62¹¹ possible tries. That comes out to about 52 million-million-million tries.

Suppose we penalize a bad answer with a time-out. Hackster is not going to stick around waiting for a response unless every response (good or bad) takes the same amount of time. Therefore, suppose our website has built-in a login delay of 2 seconds with a standard deviation of a half-second. In other words, 90% of the time a user is waiting at least a second to log in, and only 10% are waiting longer than 3 seconds (you might even truncate the distribution around those 10%-90% marks, or use a constant delay). Hackster’s machines have to wait 2-1/2 seconds or longer to be more than 70% sure that the login was unsuccessful. We stick with the worst case that Hackster picks 2 seconds because he is not as smart as I thought he was (or your delay is not random). Let us give Hackster a million threads or a million zombies or combination thereof. Since Hackster has to wait at least 2 seconds before moving his thread on to the next try, his thread/machine will wait. Well, with the bad password, Hackster is still going to win in 186 days. With that stronger password, Hackster takes over 3 million years to do the same thing!

Therefore, a reasonably strong password policy combined with a login delay of a few seconds can  mitigate a brute-force password attack, and leaks no information to a brute-force username attack as to whether the username belongs to a real user or not.

For purists, the Gaussian distribution of delay times is better, because your internal algorithms may take slightly longer one way or the other, and the Gaussian delay would tend to hide that time difference a bit better than would a constant time delay. Beyond that, you should monitor your logs and if you are under attack then start looking to see if there are IP addresses that can be blocked.

Conclusions

• If your website has a strong password policy, and pauses a second or two before responding to a login attempt (successful or not) then a brute-force attack has a very low probability of success.
• If your website has a lockout policy for known users after some number of sequential login failures, then a brute-force attack has a low probability of success. However, as username complexity decreases, the number of help desk requests to reset accounts increases and could become overwhelming.
• If your website has a lockout policy for know users, then unless the login page responds in exactly the same way to the X^th attempt of an unknown user then your website is leaking usernames to hackers.
• The lockout did hurt Hackster. After 10 failed passwords, Hackster moves on to the next username on the list. However, your phones are ringing and users are inconvenienced. The next step would be to unload the help desk by resorting to a 2^ndary failed X^th-attempt login page that asks for mother’s maiden name or presents a CAPTCHA.

The OWASP site discusses a number of solutions to this problem, and today’s blog will summarize them in a table.

First and foremost, when a prospect has a significant problem and we have the solution but they don’t buy, the first thing we should do is look in the mirror. If we don’t make the sale and we should, it’s our fault. It’s not the prospect.

For starters, too many sales people are focused on their presentation. Their presentation is the focal point of the meeting and they fail to listen and hear the problem the prospect is detailing. This leaves the prospect with that “piece of meat” feeling. They feel that all that’s important to the sales person is their own agenda.

In addition, when sales professionals focus on themselves and their agenda they miss vital details that can many times make or break the sale. Secondly, who was in control of the interview? Who asked most of the questions? Who gave most of the answers? He who gets his questions answered is in control.

The sales professional should be the one asking all the questions and the prospect should be the one doing most of the talking. If it is in reverse order, the prospect is in control. One thought to keep in mind is this: If we are spilling all of our knowledge, experience and expertise out on the floor for all to see, hear and take, we run a very high risk of unpaid consulting.

The last time I checked, I didn’t find any sales professionals that had won the noble peace prize for unpaid consulting. Yet many still do it. Last but not least, how did we behave? Did we act like every other sales person? Did we act like we needed to make a sale? Did we talk too much, launch into a presentation too fast or did we ask lame, canned, predictable questions that most likely the prospect has heard before?

The bottom line is this: everything matters. We can truly only control one thing in the prospect interview, our behavior. If we act like a sales person we will be treated like one. If we aren’t truly prospect focused, we can’t build a mutually beneficial relationship. If it is not a relationship, it is merely a transaction. There is no future in simple transactions; the future depends on relationships.

It’s not rocket science. It’s simple communication and relationship dynamics that make our sales careers either a success or a colossal failure. The choice is yours.

With the addition of the App Store in 2008, consumers were given access to thousands of games available for instant download to the mobile device.  The app market place offered both free and paid games, with prices ranging from $0.99 cents to well over $9.99. Due to the popularity of the iOS device, players experienced a revolution in casual and social gaming.

The iPad is quickly becoming a major factor in the game industry, drawing a greater number of players who have tired of the traditional game console. This progression has given rise to a new business model, “Free to play” games with an “in game store” for cash purchases. The benefit of using this model is that players can download and play a game for free, but have the option to spend real cash to improve their experience.

Using this business model, developers and publishers can draw more players to their title, since acquiring the game does not cost any money. A greater number of downloads increases the likelihood of customers investing with cash purchases. This encourages developers to create high quality products with addictive content. Furthermore, a developer can include in game advertisement to generate additional revenue.

There are many benefits to using this model, but there can also be just as many problems. The game industry has a tendency to copy designs that prove successful in the market place. As a result, the App Store is inundated with clones of popular games. This makes it difficult for a new product to stand out above the crowd, which then affects the quantity of downloads.

Another problem is that some games focus on directing players to the store at the expense of maintaining fun factor.  Gamers are intelligent and expect a rewarding experience, so if a game places too many limits on what they can do without spending money it becomes boring, and the gamer will move on. People do not want to spend countless hours grinding for currency in a game just to reach the equivalent of a $.0.99 cent purchase.

Gamers are eager and willing to spend money, but that does not mean they will blindly purchase anything in the shop. The quality of the purchases must enhance the game, and avoid becoming an endless $0.99 cent black hole to make the game more enjoyable or accessible while the user continues to pay. Producing quality gameplay and content will yield better sales if the customer can still feel rewarded without being forced to make purchases.

People tend to get hung up on pure concurrency numbers (the quantity) without assessing things like user profiles and throughput (the quality).  It’s very simple to crank up threads and, say, feed search functionality a collection of values;  sure, you’ll load the servers that way, but that doesn’t necessarily have any real relation to the number of users an application can sustain.

A better approach is to find relevant use case(s) and try to match the expected artifact throughput(s).

To illustrate, we recently completed an SAP CRM performance engagement.  The client had recently acquired additional assets and was in the process of restructuring their sales and CSR infrastructure to accommodate; everyone was being brought into the same corporate WAN, and integrated into the new CRM.  They needed multiple points-of-presence (UK, CA, US) for the test itself, they had some specific CSR ticket and sales order volumes they needed to test across varying network architectures, and they needed the resulting records to funnel through an existing multi-tiered series of webservices to their end point customers.

If we had taken a simple concurrency approach, we could have ramped up a few thousand threads against search and login, saw that servers were being loaded, and called it a day … maybe they’d have had a successful launch and maybe they wouldn’t.   What we did instead was to take a hard look at the sales/CSR volume they’d had in the past (both at the parent and at the new subsidiaries), modeled test scripts to meet those numbers with their desired concurrency, then applied that targeted load from the various geographic entry points.  We hit both the concurrency numbers and the sales order / CSR ticket volumes, hit all the key elements of their architecture, and scaled those numbers up until we found their actual bottlenecks (in this particular case, the limiting factor was at the memory configuration level in a couple of nodes … easy to find and fix if you can dial real load up and down at will, but very illusive if you have to depend on production troubleshooting).

You simply can’t find (and fix) the issues they would have encountered at launch by cranking up threads without that thread count to the artifacts created by real users.

Relating thread counts to real user counts is vital to real-world load testing.

The year 2005 saw the launch of the Xbox 360 as the first gaming machine of the seventh generation of consoles. Not only do the machines of this generation act as simple gaming consoles, they have become the centerpiece of many homes as a media hub capable of video playback, wireless internet access, and even fitness training. The technology inside these machines has become so advanced that the “Cell” microprocessor of the Playstation 3 even has applications in supercomputing.

One of the results of having so much computational power available for developers is that games today feature the most realistic graphics ever seen, incredibly deep stories, various gameplay mechanics, and highly complex AI systems. As video games become more compelling and accessible to the general public, they have become main stream and highly commercialized. Modern gamers have grown up playing games from an early age and have developed gaming intelligence.

Intelligent gamers have an understanding of what makes a game enjoyable and what features are detrimental to the experience causing them to become selective over which product is worth the financial investment. Today, developers face the difficult task of pushing the limits of the game console while trying to implement enough features to attract the modern gamer. The extraordinary cost of producing a game places a heavy burden on publishers and development studios to create a product that can compete and generate sales when it hits the market.

The video game market is inundated with products creating a highly competitive environment where the majority of games fail to sell in great quantities. The games that do the best on the market receive attention for being fully functional, feature rich, and enjoyable. Placing an emphasis on the quality assurance stage of development can help give a new product every advantage upon launching to the public.

The importance of quality assurance prior to the release of a product cannot be understated. Often times it is difficult for a developer to make changes to a game after it is released, even the occasional patch distributed online may have unexpected adverse effects. In addition, fixing games post release can be an expensive endeavor that may not pay off if the products reputation has been damaged.

When external QA testing is begun nearing the end of the product’s production cycle, it is valuable to invest a little time into exploratory testing right away. By becoming familiar with the product, the testers will have gained a context for the overall scope of their testing. With this acquired knowledge of the product and existing issues: any potential time adjustments to testing schedules will be discovered early and testing will flow smoother for the first run of test cases and requirements.

Regardless of when testing begins, with the use of excellent QA practices by skilled testers it certain that all areas of testing will be thoroughly covered. But, who would choose to be caught unprepared in a storm?

As a Quality Assurance firm with a mobile device department, we find many bugs that are related to certain devices.  A bug may be related to the screen resolution, or the use of a physical keyboard, or even the type of OS that a device manufacturer has placed over the original Android OS.  The subtle differences between Android devices must yield a careful consideration as to which devices will be used for a project.  iBeta has encountered many issues regarding screen resolution.

May it be that the application doesn’t display correctly on a 240 x 320 screen, or a tablet UI is displayed on a phone. For this iBeta would provide the developer with screenshots and log files for these types of issues.  When testing a device that has a physical keyboard, it can cause all sorts of issues with an application.  We make sure to test this form of input on all applications we test.  Almost all Android devices come with a manufacturer skin over the original Android OS, therefore it is wise to choose between a wide variety of manufacturers when selecting devices to have tested.  An application can have issues with a certain manufacturer skin, can cause conflictions with an application.

Our highly skilled test team provides usability tests based on a set goal that the client wants to achieve, while being mindful of oddities that may cause bugs on certain devices.  Any bugs that are encountered are carefully captured in the form of an official bug document as well as screenshots and crash logs when needed.  Furthermore, our large selection of mobile devices allows broad range coverage of devices that are confirmed to be supported for that application.