Tuesday, February 20, 2018

The Mobile Testing Pyramid


The Mobile Testing Pyramid


Introduction

Mobile testing involves more complexity when you compare it to web UI testing. A few examples of mobile testing challenges:
mobile testing challenges
As you can see, mobile testing not only has to cover the UI aspects, but also the compatibility of hardware, network connectivity, operation system flavors (especially on Android), etc.
When most people start with testing for mobile, they tend to start with getting a few real devices and perform exploratory testing manually. Over time, the number of devices will grow, because actual customer feedback comes in describing that their app does not work properly in certain situations on specific devices. Does this sound familiar?
In order to reach fast feedback and lower amounts of the manual testing, a better usage of available tools is necessary.

Mobile Test Pyramid

Based on my personal mobile testing experience, I came up with the mobile pyramid strategy. Of course, it was inspired by the test pyramid of Mike Cohn ('Succeeding with Agile', 2009). But this doesn't quite cover the detail that is needed for a good mobile testing strategy.
The mobile test pyramid has 3 levels:
  • Real devices
  • Mobile simulators & emulators
  • Desktop browsers (using mobile simulation)
mobile test pyramid
Identical to the generic test pyramid, the broader the layer in the pyramid, the more tests you should have which cover a wider range of complexity.
Here are the focus areas for each layer, along with their pros and cons as I see them.
Desktop browsers: mobile testing on desktop browsers
Focus areaDescription
Functional system testingIsolated browser tests performing full functional validations
Responsive designResizing browsers and toggling user agents
Cross-browserUse equivalent desktop browsers
Overall visual layoutNo extensive visual checks because the rendering is different than devices

Pros

  • Fast execution: Matter of milliseconds to launch a browser, also headless execution is possible.
  • Scalable: Easily set up 10+ browser instances per machine.
  • Cross platform: Ability to use browsers on different operating systems.

Cons

  • Mobile simulation uses the desktop browser engine: Mobile simulation in desktop browsers is still using the desktop browser.
  • No native integration: No native keyboards, incoming calls, etc.
  • Just not a device... Incredibly fast, but still not a real device.
Mobile simulators/emulators: closer to the actual mobile experience...
NOTE: Only applicable for iOS and Android.
Focus areaDescription
Functional end-user flowsClick paths throughout the application
Native API integrationGPS injection, file attachments, incoming calls etc.
Visuals (vanilla only)Use equivalent desktop browsers
Overall visual layoutEmulators are limited to vanilla versions
Touch interactionsTouch interactions such as swipe and tap comes closer to the user experience of a device than browser emulation

Pros

  • Easy to set up: Simulators/emulators are easy to set up, just download, install, and run.
  • Scalable: Virtualization means scalable and also running in parallel on one machine.
  • Native API integration: Ability to test native APIs such as incoming calls and GPS injection.
  • Simulators or Intel-based emulators are fast: Simulators are fast, because they only have to simulate the software part. Emulators based on the Intel architecture are fast.
  • Debugging possibilities: Easy to debug simulator/emulators, already hooked up to the machine to access logs.

Cons

  • Vanilla versions only: Manufacturer’s skins are available, but the device behavior is still based on what comes stock.
  • No real resource usage: CPU/memory usage of machine in case of simulators. Emulators try to simulate the hardware.
  • No real interoperability: Connectivity with NFC, Bluetooth, network connections.
  • Slow ARM-based emulators: Emulators based on the ARM architecture are slow, which is the main architecture for Android devices.
  • Inaccurate color display in light/dark: Contrast/brightness inaccurate in light/dark environment.
Real devices: the real thing...
Focus areaDescription
UsabilityValidating usability such as actual click areas, touch actions and voice over
PerformanceCPU/memory usage, battery, network strengths
Native API integrationInterruption (incoming calls, push notifications), resource fighting (camera, GPS), NFC, Bluetooth
VisualsFocus on devices which are not available as simulators/emulators
Manufacturer’s sauceReal OS from manufacturers, e.g. Samsung's TouchWiz and built-in browsers

Pros

  • Native APIs in real conditions: Ability to test native APIs not only with injections for automation, but also actual NFC touch for example.
  • Can be faster than emulators: Some real devices are just faster than emulators due to the simulation of hardware, especially compared to the ARM-based emulators.
  • Just the real thing... Actual network conditions, battery/CPU/memory usage, manufacturer’s secret sauce on top of the OS.

Cons

  • Costs: Real devices come with a price, usually you pay per device/cradle.
  • New device means procurement: A new device is usually not available on-the-fly, even with cloud solutions. E.g. when the new iPhone comes out, it's not available immediately to procure. In the meantime, the iOS simulator would already be available.
  • Development iOS build required for automation: iOS apps need to be signed with Development Distribution Certificate and Provisioning Profile for automation.

Conclusion

There are loads of trade-offs when it comes to mobile testing. But by learning to use all layers of the mobile testing pyramid to your advantage (leveraging desktop browsers, mobile simulators/emulators, and real devices together) you can gain the fast feedback that is required in modern CI/CD environments. It's just a matter of focusing on the right things in each layer for your context.



Cheers,
Naveen AutomationLabs

Monday, February 19, 2018

Test Cases - Game Testing - By Naveen AutomationLabs

Test Cases - Game Testing



Mobile Game Testing
Check for background music
and sound effects
ON/OFF sound & background music
Receive the call and check
Verify if sound effects are in sync with action
ON/OFF device sound(native sound) and check
Check for vibration effect if present
User InterfaceCheck in Landscape/Portrait mode
Check for animation, movement of character, graphics, Zoom In/Out (all gestures) etc
There should not be any clipping (cutted background)
Test whether one object overlaps with another
Verify if loading indicator is displayed wherever required
Character should not move out of the screen/specified area
Test for enable and disable images/icons/buttons etc
Check for screen title
Check for message title, message description, label (should be appropriate)
Check scrolling
Font displayed (color, size etc)
Check other objects too (ex -if its a car race- you need to look at road, people, other objects like buildings etc)
PerformanceCheck the loading time of a game
Make sure that any action is not taking considerable time, game flow should be fast
Scorescore calculation
Verify leaderboards General/All time/Weekly/local etc
Check the score registration functionality
Check the format (whether, comma is required in score etc ideally if customer is a foriegner coma should be in millions not in thousands )
Check for level completion syncs with the score
Time OutCheck for time out
Do the actions when time-out yet to happen
MultitaskingSwitch b/w different apps and play game , check for sound, score, UI, time-out etc
PauseCheck if game is paused when call received or multitasking or sleep mode
Save SettingsTurnoff and ON device, check if settings are saved
Log out /On , check same
User should not loose his game in above conditions
User profilePut a all types of images in Player profile and check
Put special character, numbers,space in username and check
Password should be in masked
Chat featureCheck the profile images
max limit of chat description
Enter empty string, special character and check
For a opponent , there should be a notification that he has received a message
FunctionalityCheck game area, game logic
play till last level
get the cheat codes from development team and check all the levels
Check for the features that will be unlocked level-wise
Check for bonus score
Check the score hike when level gets increased
Check for multi-tap action (example in a car race we hold accelerator and left/right turn button simultaneously)
Menu options
Different game modes/location
Help & About ScreenShould be in easily understandable format
free from spelling mistakes
URL should be hyperlinked (depends)
Multiplayer gameSession expiry check
login/log out
Registration (Sign Up)
Verify account (receive verification mail)
login with registered but not verified account (without clicking verification link)
Forgot password checks (many cases here)
Game flow
Check for WIN/lost/Draw
Check user statistics graph
Challenge/Decline challenge/receive challenge
Check for forfeit
Check when player 2’s turn is on Player 1 is not able to do actions (should not be able to forfeit also)
Check for pass turn
Check for time-out (for one player)
Check the score for both the players till game ends
Memory leakCheck the game when device memory is low
Network (n/w) checkN/w messages if n/w is not present
check if what happens when n/w not present and user plays a move (whether score submitted for that move etc)
Check for localizationShould be Support of different languages
Check for time formatChange the device time , format etc
SizeUser wont like if your game takes lot of device space, so keep one eye on game file size
Device , OSCheck in supported screen sizes and os versions (basicaly depend upon Client requirement)
Depends on platformSometime we need to check as per OS guidliness as well. For ex in Wp7 we need to check in 2 background (light/dark).
Check Share optionsPost score via mail/FB/Twitter
Check the posted/sent messages in FB/Twitter/Mail. Check links are hyperlinked and application icon is displayed in
the post (depends)
If twitter integration is a manual ( custom UI developed by developer), check what happens when u enter more than
140 chars (as twitter limit is 140)
Music playing during app
launche
If music player running and we start installing any game app, music player should pauses without prompting for the user permission.
Steps for checking this:
1. Play a music file.
2. Launch the application.
3. Verify that while the application loads, it does not pause, resume or stop the actively playing music.
InteruptionIf app(game) is in running mode, then Check the behaviour of interuption like like Bluetooth, Infra red and CALL/SMS/MMS.
Upgrade the game
/ Battery effect
Upgrade of Games to the latest version and while migration all data should persist [ score, user profile etc ]
What if Battery goes down/switched of the cell while playing, Wheter the score wil get saved?

Sunday, February 11, 2018

Web Services - API Automation Tutorials

Web Services - API Automation Tutorials


Hello Guys,
~~I'm launching my first video web series on Web Services - API automation on Vimeo. These videos are totally based on API - WS automation using HTTP Client and Rest Assured with framework designing step by step.

~~Interested people can buy these video directly on Vimeo. You will get life time access on these videos and will get full access on all upcoming videos at free of cost. Here is the link:



Course Content: 1. Manual Testing of WebServices/API: Introduction of API and WebServices: What is API What is WebService What is Backend Architecture Rest vs SOAP APIs What is CRUD operation What are different HTTP Calls – GET/POST/PUT/Delete Different Live Projects Examples 2. Postman: Introduction How to call Rest API in Postman How to pass path parameter in Request How to pass query parameter in Request How to set Headers in Postman How to pass JSON/XML Payload How to check response status code How to check JSON/XML response message What is response header 3. Rest Client and Advance Rest Client: Introduction How to hit REST APIs in Rest Client Basic and Advanced Settings in Rest Client How to check APIs at the network layer using developer tools of browser: How to check Backend services – APIs running behind What is developer tool in Chrome How to check request/response of any API in Firefox/Chrome browsers 4. What are different HTTP Status Response Codes: 100 series 200 series 300 series 400 series 500 series 5. Automation Testing of Back End Services: Learn HTTP Client: All CRUD – GET/POST/PUT/Delete Calls How to send Request with Payloads What is JSON Payload How to validate JSON response using JSON Parsers Common Utilities 6. Learn Rest Assured API in Java: Introduction Rest Assured methods: GET/POST/PUT/Delete BDD Framework in Rest Assured 7. API functional Testing using Jmeter Tool (only functional part) 8. API functional Testing using SOAP UI Tool 9. Maven: Build Automation Tool 10. TestNG (TDD) Framework for API automation 11. Cucumber (BDD) Framework for API automation 12. How to run API automation using Jenkins – CI tool (Continuous Integration) 13. Interview Questions: How to crack Product Companies interviews 14. Life Time Free Access on all Videos/Recordings

https://vimeo.com/ondemand/webservicesapiautomation


Cheers! Naveen AutomationLabs

Featured Post

Advanced Selenium Webdriver Interview Questions and Answers in Java

Advanced Selenium Webdriver Interview Questions and Answers in Java Selenium WebDriver - QA Here is the list of Selenium Inte...