How Do I Convert This Selenium Next Button Clicking Code Into Loop So I Can Get Url Of All The Pages Until Next Button Disappears

How do I convert this selenium next button clicking code into loop so I can get url of all the pages until next button disappears

Contents hide

1 How do I convert this selenium next button clicking code into loop so I can get url of all the pages until next button disappears

2 Solution – 1

3 Solution – 2

Problem Description:

Hello I wrote this selenium code to click Next button and give me url of the next page.

The Question is

I want to convert this code in a loop so I can click on the next button & collect all URLs until next button disappears.
How do I out all the collected URLs in a list?

next = driver.find_element(By.LINK_TEXT, "Next")
next.click()
urrl = driver.current_url
print(urrl)
driver.quit()

I tried While True loop for this.

while True:
    try:
        urrl = driver.current_url    **## I tried this line after clicking the next button as well**                       
        next = driver.find_element(By.LINK_TEXT,"Next")
        next.click()
    except:
        break

I was able to click on the next button until the end but I can not figure out how to collect url of the webpage and how to append them into a list.

Tried append but I think I am doing something wrong.

Solution – 1

You can write a function to test if the element exists:

def is_element_exists(xpath, id_flag=False):
    try:
        if id_flag:
            driver.find_element_by_id(xpath)
        else:
            driver.find_element_by_xpath(xpath)
        return True
    except Exception as e:
        # print("Excpetion:[%s][%s]" % (e, traceback.format_exc()))
        print('do not find the node')
    return False

Solution – 2

You can define a list object and append the collected URLs there as following.
The list should be defined before and out of the loop.

urls = []
while True:
    try:
        urrl = driver.current_url
        urls.append(urrl)
        next = driver.find_element(By.LINK_TEXT,"Next")
        next.click()
    except:
        break
print(urls)

The code above is generic. Probably you will need to scroll to the "next" button and wait for it to become clickable etc.

Rate this post