Python Scraping Can't Get Data

The page contains several hidden parameters (for example, __EVENTVALIDATION).Obtain the values of these parameters on the first access and submit the form data.

Below is the python script that does this, but the response (HTML file) contains only the first 50 search results.To get all the search results, you need to add a link to each page to retrieve the HTML file.

import urlib
from bs4 import BeautifulSoup

# first access —get hidden parameters
url=r'https://nintei.nurse.or.jp/certification/General/(X(1)S(efl0y555pect3x45oxjzfw3x))/General/GCPP01LS/GCPP01LS.aspx'
html=urllib.request.urlopen(url)
soup = BeautifulSoup(html, 'lxml')

# generate form data
count=int(soup.select('#__VIEWSTATEFIELDCOUNT')[0]['value'])
form_data = {
  '__VIEWSTATEFIELDCOUNT': count,
  '__VIEWSTATE': group.select('#__VIEWSTATE') [0] ['value'],
  '__EVENTVALIDATION': group.select('#__EVENTVALIDATION') [0] ['value']
}
for i in range (1, count):
  form_data [f'___VIEWSTATE{i}'] = group.select(f'#___VIEWSTATE{i}') [0] ['value']

form_data ['ctl00$plhContent$btnSearchMain'] = 'Search'
form_data ['ctl00$plhContent$drpField'] = -1
form_data ['ctl00$plhContent$drpNameOwnerWorking'] = -1
form_data ['ctl00$plhContent$drpWorkPrefecture'] = -1
form_data ['ctl00$plhContent$drpWorkType'] = -1
form_data ['ctl00$plhContent$radlstCert'] = 1

# second access:get search result
form_data = urllib.parse.urlencode(form_data).encode()
html=urllib.request.urlopen(url,form_data).read().decode()
print(html)

There are many different ways of holding/showing information on a website, and there is no such thing as a template that can be applied to different sites, regardless of whether it is created by the same company for the same purpose.

Forget the idea that some template exists (applicable) and start by looking at the content/configuration of the target site.

• On the page, it seems that all information can be obtained at the initial value unless otherwise specified.
  By specifying the categories of Certified Nurses, Certified Nursing Managers, and Professional Nurses, you will be able to get everything.
• However, there are only 50 views each, so you need to click the Next button to proceed.
  The source provided is looped with a list of a tags, and you click the "Find" button in it, but it does not match the content of this page.
• Therefore, the how to get all of these at onceis probably not.
  Currently, there are 2,072 registered "certified nurses" and 415 pages of loops are required.
  There is a possibility that it can be retrieved at once in some way, but it is probably private and not always available.

Note:
Full data published officially: (available on Excel sheet)
Information Processing Security Advisor Search Service

Examples that can be retrieved by behind-the-scenes techniques:
(The comments show behind-the-scenes techniques) Python scraping table cannot be retrieved
I am also responding to the reference information for obtaining it by scraping.
Slape with selenium (next page available)

Here's how you think.

• base='https://nintei.nurse.or.jp/certification/General/' and later review
• First time driver.find_element_by_xpath('//*[@id="ctl00_plhContent_btnSearchMain"]Get it with ').click()
• Use Selenium or BeautifulSoup to extract data with XPath in '//* [@id="ctl00_plhContent_dlvMain"] (use it if there's another way you think it's easy to use)
  The links to the front and rear pages and item names are also included in the table, so if you don't need them, I'll shave them off
• Next button click
using driver.find_element_by_xpath('//*[@id="page_navi_wrapper"]/ul/li[9]/a').click()
• If you try to display it to the end and go further, there will be no element and it will be an exception, so check for element in advance or exit the loop by detecting an exception in try:exception: