How to filter fows in Pandas using partial string in column

How to filter fows in Pandas using partial string in column

Problem Description:

I’ve used the SerpAPI to pull down some data about jobs in a sector I want to return to.

There is a lot of junk about training and I’d like to remove the results based on the displayed_link column.

position    title   link    displayed_link  date    snippet snippet_highlighted_words   sitelinks   about_this_result   about_page_link about_page_serpapi_link cached_page_link    related_questions   rich_snippet    related_pages_link  thumbnail   duration    key_moments
0   1   What Does a Data Analyst Do? Your 2022 Career ... › Coursera Articles ›...   Nov 14, 2022    A data analyst is a person whose job is to gat...   [data analyst]  {'inline': [{'title': 'Business analyst', 'lin...   {'source': {'description': 'Coursera Inc. is a...   NaN NaN NaN NaN NaN NaN
1   2   What Does a Data Analyst Do? Exploring the Day... › degrees › technolo...   Sep 19, 2022    Generally speaking, a data analyst will retrie...   [data analyst, Data analysts]   {'inline': [{'title': 'Where Do Data Analysts ...   {'source': {'description': 'Rasmussen Universi...   NaN NaN NaN NaN NaN NaN
2   3   Become a Data Analyst Learning Path - LinkedIn › learning › become-a...   NaN Data analysts examine information using data a...   [Data analysts, data analysis]  NaN {'source': {'description': 'LinkedIn is an Ame...   NaN NaN NaN NaN NaN NaN NaN
3   4   What Does a Data Analyst Do? - SNHU › about-us › newsroom › stem   

Tried manually creating of the sites I want to exclude sites in this list

promotions = ["coursera"

Tried this:

df['displayed_link'].map(lambda x: "T" if x in promotions else "F")

And all it does is return F – I’m guessing because it needs exact string.

df['displayed_link'].map(lambda x: "T" if promotions in x else "F")

I tried it the other way, but that was a syntax error.

What is the most efficient way of filtering rows based on a column based on a list of manually curated strings?

    enter code here

Solution – 1

Use Series.str.contains with chain list by | for regex OR:

df['test1'] = np.where(df['displayed_link'].str.contains('|'.join(promotions)), 'T', 'F')
df['test2'] = (df['displayed_link'].str.contains('|'.join(promotions))
                                   .map({True:'T',False: 'F'}))

If necessary, use words boundaries bb:

pat = '|'.join(rf"b{x}b" for x in promotions))
df['test3']= np.where(df['displayed_link'].str.contains(pat), 'T', 'F')
df['test4']= df['displayed_link'].str.contains(pat).map({True:'T',False: 'F'})
print (df)
                                  displayed_link test1 test2 test3 test4
0     T     T     T     T
1     T     T     T     T
2     F     F     F     F
3     T     T     F     F
Rate this post
We use cookies in order to give you the best possible experience on our website. By continuing to use this site, you agree to our use of cookies.