Should we check static UI elements with Automation Tests

Recently, I came across a scenario where all the Xamarin forms for the app had an update. This resulted in issues like static text being cut off, not being visible, changes in colors for a few buttons’ state, etc on different kinds of devices.

Will automation catch such changes?

Before we conclude, let’s pause and reflect - “Is this the purpose of automation?” “Do we write automation to detect such issues?”

Here are some reasons for testing static content on a wide range of device sizes :

• How do you test all the static text on multiple devices with different screen sizes/resolutions? Is it practical by automating?
• How much maintenance will it require?
• How much is the risk of false positives? (I am talking about just the English text here, no localization. Localization logic is at a lower layer. Testing Localization code is best suited at lower layers of the test pyramid. Localised content would in any case be reviewed by specialists.)

Will you say that automation is not smart enough? Or are we failing to automate such checks correctly?

I ask - Is there a need?

My Thoughts :

Can we take care of static text checks at the time of development? I suggest this has to happen in-person between designers and developers whenever there are form factor changes or layout changes. The Designer would be a stakeholder who would review the changes on the relevant form factors and accept the changes. Thus, we could have a specific check before the changes are pushed to the production environment.

Why do we automate? Is it just because we can?:

“We automate because we can” - This is a common interpretation of automated test design. Let’s ponder a little on this interpretation:

After automating the static text check, and even if half of the text is not visible, or cut off, or moved below the viewport, the test case is going to pass as the element has all the content.

But then, that UI element that just cleared the test is in fact just not visible to the eyes of the actual user!

I can check for the corners of the box or if the text is outside of the box, but is that what automation is for? What is the aim of automation?

Automation check of static text should comprise the following:

• In automation, you should just read the basic text required to navigate to the next workflow. Text e.g. - submit, login, next, view, download, etc.
• For regression/smoke - I still should not be asserting on the static content/text. Such checks should be at lower layers of the test pyramid.

Automation should stick to - checking that UI has received the data, finding the controls and their working condition and clicks to continue in the workflow or to move to the next workflow. This would help separate a rendering check from a functionality check.

How to verify content (static text)? :

• If there is content that you want to ensure is available - perform lower-level text verification in the code that generates content.
• Review rendering & layout fixes manually.
• A One-time manual review can achieve a lot more in less time as compared to automating it.
• Once the layout is satisfactory, review it only when the layout or the static text changes.
• It is important to verify messages/static text not just in isolation but in the context of the rest of the UI and who other than a human is best suited for this?
• Because the errors in this case you often see have to do with the layout of the page and/or the correct rendering of the character sets, which are difficult to test in an automated way.

There are tools in the market for checking the sizes, shapes, text, colors etc. - but matching may fail because of external reasons. While you could ram up your skills, you would then be depending upon the correctness of that Visual Check tool, and would need to assess it. One time review as part of story review would remove the need for a visual check tool.

Developers can do this themselves while developing and before pushing a commit for the central build. The designers are the stakeholders who are best positioned to review the form factor changes.

We have Acceptance criteria as part of the user story to determine what checks we need for that content or form factor change and the responsible stakeholder for the same.

A static item won’t change between automation runs on a fixed form factor. There is thus no point wasting automation capacity on static item verification.