Home:ALL Converter>Google Scraping Results Difference - Google Custom Search API vs Google Selenium Search

Google Scraping Results Difference - Google Custom Search API vs Google Selenium Search

Ask Time:2021-07-24T13:30:57         Author:SY Moon

Json Formatter

I wrote two scraping codes for Google search results.

Google web-site chrome search (selenium) / Google custom API search (free, 100 quires limit)

When two codes are executed, the conditions are the same but the results are different(Maybe I am wrong?)

Google Web site selenium "url"

GoogleUrl = f'https://www.google.com/search?q={SearchKeyword}+intext:{MUSTKeyword}&hl=en&rlz=&start={pageNum}'

This is the Google API parameter that I use.

response = service.cse().list(
        q=searchterm_in,
        exactTerms=incl_searchterm_in,
        siteSearch=sitesearch_in,
        cx=my_cse_id,
        num=10,
        start=index,
    ).execute()

I thought "search keywords" = "q", "MUSTKeyword" = "exact terms".Google API doesn't seem to have "intext:" parameters in the official document.(https://developers.google.com/custom-search/v1/reference/rest/v1/cse/list), So I tried the "exactTerms" but I was able to get the results more accurately with the Selenium Google search results than the API.

Question: How do I set or add parameters in the Google API to import the same results into Google Selenium Search? What replaces "intext:" in Google API search parameters?

Author:SY Moon,eproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/68507258/google-scraping-results-difference-google-custom-search-api-vs-google-selenium
yy