Introduction
If you’re interested in a better solution to this problem, take a look at my follow up blog on this specific subject! There we utilize the
shadow_root
attribute as it was designed to be used to achieve the same goal.
When you’re developing software to last, building tests along the way is integral in maintaining a robust inflexible code-base. ServiceNow’s ATF is an efficient, cost-effective way of doing regression testing in the ServiceNow platform, but it missed the mark on navigating Custom UI pages as well as third-party integration testing. That’s where using Selenium come’s in. Selenium is an automated testing tool that multiple languages and multiple browsers, allowing developers to create tests by navigating the page and integrate those tests using frameworks such as TestNG and JUnit.
Feature | Support |
---|---|
Programming Languages | Java, Python, C#, Ruby, JavaScript, etc. |
Browsers | Google Chrome, Mozilla Firefox, Safari, Internet Explorer, etc. |
Integrations | TestNG, JUnit, Jenkins, Docker, Maven, etc. |
We set out taking on the simple task of logging in, navigating from the home page to the Incidents table and logging out using just Selenium in order to tackle one of the biggest hurdles of Selenium in ServiceNow, the Shadow DOM.
Understanding Shadow DOM
Shadow DOM is a web development technique that provides a way to encapsulate or “hide” elements of a web page’s Document Object Model (DOM), offering a level of separation between an element’s functionality and style from the rest of the page. This encapsulation makes it possible to keep an element’s features private, allowing developers to style and script elements without interfering with other elements on the page. However, it introduces challenges of compatibility with Selenium’s testing library. The elements that are “hidden” within the Shadow DOM are difficult to identify with traditional Selenium methods.
Difficulties of Selenium in the Shadow DOM
Using Selenium, you can use use the find_element()
function to grab practically any element. It gives you the option of choosing unique identifiers of said elements such as CSS, Name, ID, ClassName, XPATH, etc. This is always worth trying first, but with elements that are within the ShadowDOM it often falls short in being able to identify and acquire the obfuscated element.
![Using Browser DevTools to grab the xpath of the 'all' element Browser DevTools, grab XPath, 'all' element, web development, web testing, inspecting elements.](https://www.seamlessmigration.com/wp-content/uploads/2023/06/20230616_152205_image-1.png)
![Copying the XPath of 'All' element: XPath, 'All' element, copy XPath, browser DevTools, web testing, web development.](https://www.seamlessmigration.com/wp-content/uploads/2023/06/20230616_152225_image.png)
Below is the traditional method to grab an element using Selenium using find_element
:
all_button = snTest.driver.find_elemen
t(By.XPATH'//*[@id="d6e462a5c3533010cbd77096e940dd8c"]')
Plus, Selenium has provided the shadow_root()
function to give user’s the identify the shadow DOM
This method, however, provides little help as it doesn’t give the developer any real way to interact with the Shadow DOM, and, in the case of ServiceNow, there are multiple shadow roots nested on top of each other. It becomes very complex, trying to identify and interact with each shadow root to get to the element you’re searching for.
Using JavaScript with Selenium
In order to interact with elements inside the shadow DOM, we had to explore the only thing that could touch them: JavaScript. Since Shadow DOMs are a browser standard, they can be interacted with using JavaScript directly. After some experimentation and learning, we found that we could execute JavaScript code directly from Selenium.
The solution was to utilize Selenium’s execute_script()
function to invoke JavaScript code that could access and interact with elements inside the Shadow DOM. We crafted custom JavaScript paths to locate and interact with shadow elements, effectively ‘bypassing’ the encapsulation to access the elements we needed for our tests. To acquire the JavaScript path of an element, it’s as simple as using DevTools and copying the JS path directly.
![Copying JS Path: Copy JS path, browser DevTools, JavaScript, web testing, web development.](https://www.seamlessmigration.com/wp-content/uploads/2023/06/image-1.png)
Using this method provides the developer with JS script using querySelector()
functions to traverse the shadowRoots until they’ve reached the desired element. From this point, you can call whatever JS function you’d like, including a simple .click()
and it will be returned in the execute_script()
function.
![Example of JS path given by DevTools: DevTools, JavaScript path, inspecting elements, DOM elements, web development, web testing.](https://www.seamlessmigration.com/wp-content/uploads/2023/06/20230616_154537_image.png)
Overcoming the time.sleep() Dilemma: Reintroducing Selenium Explicit Waits
Initially, time.sleep()
was used as a quick workaround to deal with latency issues. It was an easy solution for letting the page load or waiting for elements to become visible. However, it soon became clear that this was not a scalable or reliable solution. time.sleep() introduces arbitrary wait times which can make the tests slower than necessary and may lead to false negatives if the wait time isn’t long enough.
To overcome this, we implemented an explicit wait strategy using the is_element_present()
function. This method checks for the presence of an element each time it is called. When coupled with Seleniums explicit wait function WebDriverWait()
we can still verify that the element exists by continually polling the is_element_present()
function. This strategy ensures we wait only as long as necessary for an element to appear, increasing test efficiency and reliability.
![is_element_present() function: Selenium WebDriver, JavaScript, DOM elements, detecting element presence, exception handling.](https://www.seamlessmigration.com/wp-content/uploads/2023/06/isElementPresent.png)
Selenium/JavaScript Solution
The click_shadow_element
function in the ServiceNowSelenium
class is an example of how we implemented this solution. It uses JavaScript to wait for and click a shadow DOM element based on the JavaScript path provided.
![Utilizing is_element_present() usage with explicit waits: Selenium WebDriver, JavaScript, shadow DOM elements, is_element_present function, explicit wait method](https://www.seamlessmigration.com/wp-content/uploads/2023/06/image-2.png)
ServiceNowSelenium
Class:
from selenium import webdriver from selenium.webdriver.common.by import By from selenium.common.exceptions import NoAlertPresentException, UnexpectedAlertPresentException, TimeoutException from selenium.common.exceptions import ElementNotVisibleException, ElementNotInteractableException from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.common.exceptions import TimeoutException class ServiceNowSelenium: # constructor # contains driver, url and login credentials def __init__(self, url, username, password): self.driver = webdriver.Chrome() self.url = url self.username = username self.password = password # Login function, initializes driver and logs in based on values provided def login(self): # Initialize self.driver.maximize_window() # Full screen window self.driver.get(self.url) # Navigate to url given by constructor self.accept_alert() # Catch any alerts # Login using username/password username_field = self.driver.find_element(By.NAME, 'user_name') username_field.send_keys(self.username) password_field = self.driver.find_element(By.NAME, 'user_password') password_field.send_keys(self.password) login_button = self.driver.find_element(By.ID, 'sysverb_login') login_button.click() # Logout and quit driver function def logout(self): user_menu_path = '''[copy/paste js path]''' self.click_shadow_element("user menu button", user_menu_path) logout_path = '''[copy/paste js path]''' self.click_shadow_element("logout button", logout_path) self.driver.quit() # Check if an element is present using JS path. # Particularly useful for shadow DOM elements def is_element_present(self, js_path): try: element = self.driver.execute_script(f"return {js_path}") return element is not None except Exception: pass try: outer_html = self.driver.execute_script(f"return {js_path}.outerHTML;") if outer_html: return True;<br /> except Exception: return False # used to click shadow elements w/ js path. # uses javascript to allow for explicit waits def click_shadow_element(self, errorName, js_path, wait_time = 10): try: WebDriverWait(self.driver, wait_time).until(lambda x: self.is_element_present(js_path)) self.driver.execute_script(f"return {js_path}.click()") except (TimeoutException, ElementNotVisibleException, ElementNotInteractableException) as e: raise Exception(f"The {errorName} button was not found. Additional info: {str(e)}") # Wait and accept alerts as they come def accept_alert(self): try: WebDriverWait(self.driver, .5).until(EC.alert_is_present()) alert = self.driver.switch_to.alert alert.accept() print("Alert accepted") except (NoAlertPresentException, TimeoutException, UnexpectedAlertPresentException): pass
Key Learnings and Insights
Handling Shadow DOMs with Selenium proved to be an enlightening challenge. We learned that traditional Selenium methods may fall short when it comes to web components and Shadow DOM. However, we found that Selenium’s ability to execute JavaScript provides a powerful tool to interact with these encapsulated elements.
Furthermore, we learned the importance of implementing efficient wait strategies for page and element loading. The decision to replace time.sleep() with explicit waits was critical for improving the speed and reliability of our test.
Future Directions and Concluding Thoughts
Moving forward, it would be worthwhile to continue refining our JavaScript interaction techniques, making them more robust and adaptable to different web component structures. Additionally, investigating other testing frameworks, libraries, or tools that offer built-in Shadow DOM support could be beneficial.
In conclusion, navigating Shadow DOM with Selenium in ServiceNow was a complex task that required creative solutions and a deeper understanding of both the tool and the web technologies involved. The journey underscored the importance of adaptability in automated testing and provided a richer perspective on handling web components. Despite the challenges, the experience was rewarding, deepening our understanding of automated testing’s intricacies and nuances.
Using the ServiceNowSelenium to Run A Test:
Login, Navigate to Incidents Table and Logout: