I'm trying to load the contents of a page in a site using ruby's open-uri. 403 error (accessible via browser).
url="https://www.hogehoge.jp"
opt = {}
opt ['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36'
opt['ssl_verify_mode'] = 'OpenSSL::SSL::VERIFY_NONE'
opt['Accept-Encoding'] = 'deflate'
opt['Accept-Language'] = 'ja, en-US; q = 0.9, en; q = 0.8'
page=open(url,opt).read
Development is cloud9 and production is aws lightsail.
When a 403 error occurred during development, I thought cloud9 was not for overseas IP, so
When I installed a proxy server on my own domestic server using squid and accessed it,
I was able to connect without any problems.
However, when I deploy to lightsail using capistrano, I get 403 error.
I set the same proxy for the lightsail environment variable.
Of course, the contents of the application should be the same as the IP and http headers from which it was accessed. I can't think of any reason for the 403 error on one side.
Thank you for your cooperation.
[Additional note]
You can view the browser by viewing
We're sorry, but something wrong.
If you are the application owner check the logs for more information.
It's the one that comes out a lot.
The production.log is as follows:
I, [2018-02-02T12:46:56.074182#2970]INFO --- Started GET "/xxxx/getdata" for xxx.xxx.xxx.xxx at 2018-02-02 12:46:56+0900
I, [2018-02-02T12:46:56.075338#2970] INFO --:Processing by xxxxController #getdata as HTML
I, [2018-02-02T12:46:56.267191#2970] INFO --—Completed 500 Internal Server Error in 192ms (ActiveRecord: 0.0ms)
F, [2018-02-02T 12:46:56.267782#2970] FATAL --:
F,[2018-02-02T12:46:56.267829#2970]FATAL --:OpenURI::HTTPError (403Forbidden):
F, [2018-02-02T 12:46:56.267853#2970] FATAL --:
F,[2018-02-02T12:46:56.267881#2970]FATAL --:app/controllers/xxx_controller.rb:25:in`getdata'
[Additional note]
When I changed to net/http retrieval method, I now get 403 error when I put it in the job file, trying to put it on the controller and delaying it.
parsed_url=URI.parse('https://www.hogehoge.jp')
https=Net::HTTP.new(parsed_url.host, parsed_url.port)
https.use_ssl=true
requ=Net::HTTP::Get.new(parsed_url.request_uri)
res = https.start do | x |
x.request(req)
end
page=res.body.force_encoding("UTF-8")
[Additional note]
The response header and response body were obtained using the following methods:
The varnish cache server appears to be returning an error, but I don't know why.
logger.error("***** response header*****")
res.each_header do | name, val |
logger.error("name=#{name},val=#{val}")
end
page=res.body.force_encoding("UTF-8")
logger.error("*******page*******")
logger.error(page)
Results
E, [2018-03-26T15:35:43.256675#8988] ERROR --:***** response code*****
E, [2018-03-26T15:35:43.256720#8988] ERROR --:403
E, [2018-03-26T15:35:43.256742#8988] ERROR --:*****response header*****
E, [2018-03-26T15:35:43.256766#8988] ERROR --:name=server,val=Varnish
E, [2018-03-26T15:35:43.256785#8988] ERROR --:name=retry-after,val=0
E, [2018-03-26T15:35:43.256803#8988] ERROR --:name=content-type,val=text/html; charset=utf-8
E, [2018-03-26T15:35:43.256819#8988] ERROR --:name=content-length, val=421
E, [2018-03-26T15:35:43.256836#8988] ERROR --:name=accept-ranges, val=bytes
E, [2018-03-26T 15:35:43.256857 #8988] ERROR --:name=date,val=Mon, 26 Mar 2018 06:32:14 GMT
E, [2018-03-26T15:35:43.256876#8988] ERROR --:name=via,val=1.1 varnish
E, [2018-03-26T15:35:43.256893#8988] ERROR --:name=connection, val=close
E, [2018-03-26T15:35:43.256909#8988] ERROR --:name=x-served-by,val=cache-nrt6149-NRT
E, [2018-03-26T15:35:43.257132#8988] ERROR --:name=x-cache,val=MISS
E, [2018-03-26T15:35:43.257152#8988] ERROR --:name=x-cache-hits,val=0
E, [2018-03-26T15:35:43.257168#8988] ERROR --:name=x-timer,val=S1522045935.830867, VS0,VE0
E, [2018-03-26T15:35:43.257185#8988] ERROR --:****** page********
E, [2018-03-26T15:35:43.257201#8988] ERROR --:
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd" >
<html>
<head>
<title> 403 Forbidden</title>
</head>
<body>
<h1>Error 403 Forbidden</h1>
<p>Forbidden</p>
<h3>Guru Mediation:</h3>
<p>Details:cache-nrt6149-NRT1522045935 1982824828</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
It has been rejected for some reason, but OpenURI cannot get a response body when an exception occurs, so we don't know the details.The same is true because logs such as proxies show only the response code.
The response body may contain details, but you will need to rewrite the code to something like net/http instead of OpenURI.
The answer is to check the sub-status code, but it's unique to IIS and I can't tell by looking at the logs.
Does the error log contain a substatus code?Look in the log near the string "403".
■ IIS status codes are decimal places page has a log example, so please refer to it.
Knowing the substatus code for the 403 error (https://ja.wikipedia.org/wiki/HTTP_403) will narrow down the cause of the error.
© 2024 OneMinuteCode. All rights reserved.