Error in compiling RStudio calculated table using xtable function

What should I study first to solve this problem?

If this is the purpose of your question, the answer is simple.I think it's better to learn how to use LaTeX, XeLaTeX, LuaLaTeX, etc. that corresponds to Japanese before entering the R code.

If you can sort out whether it's an R/RStudio problem, LaTeX general problem, or a problem specific to Japanese LaTeX, you'll be more likely to find your own answer, and even if you don't know it yourself, you'll be more likely to get it.It lowers the hurdle of the respondent and makes it easier to reach the right community.

documentclass is a problem specific to RStudio, but compiling using platex+dvipdfmx is troublesome if you want to complete it on RStudio, so you should avoid the jsarticle class.

Output the table calculated by R using the xtable function and
by TeX I thought about compiling it.Print out the TeX code below and paste it into Source as shown in 2. Compile

There is a better way to copy tables generated by R to LaTeX sources.

In any case, you will learn about the knitr package.If you don't need the full power of LaTeX composition, you can learn how to write R Markdown by referring to here.Output to other formats such as HTML output is also supported.

Personally, I use RSweave (Rnw format) when I use complex compositions, and R Markdown when I use simple compositions (often so) while looking up the pandoc options.

In order to put a table generated by R in a Japanese document in R Markdown, create a file similar to the following and save it with the extension .Rmd.If you open it in RStudio, you should see the Knit button, so pressing it should generate a mixed PDF of Japanese and R output (http://ipafont.ipa.go.jp if the IPAex font is not installed).

---
title:sample
author —Name.
date: '`r format(Sys.time(), '%Y/%m/%d')`'
output:
  pdf_document:
    config_caption:yes
    keep_tex —no
    latex_engine —xelatex
    number_sections —Yes
    template:null
    toc —no
documentclass —bxjsarticle
classoption: 
  - xelatex
  - ja = standard
---

``{r,include=FALSE}
options(xtable.comment=FALSE)
```

# Table Output

``{r, echo=FALSE, results='asis'}
# write the code to generate the table here
# In this sample, the pressure data built into R.
xtable::xtable (pressure)
```

Looking at the error message, do you feel that the processing system you are trying to use does not correspond to the contents of the file?
When I read the question, it seems that RStudio compiled it with XeLaTeX, so please change the documentclass from jsarticle to bxjsarticle.

Also, the original code

I need to correct the two points in , but the output of xtable will be properly shaped here, so I think it's a copy error when posting.

This is the output of the xtable.

%latex table generated in R 3.3.2 by xtable 1.8-2 package
% Fri Jul 07 00:45:59 2017
\begin {table} [ht]
\centering
\begin {tabular} {rrr}
  \hline
 &ans & p.ans\\ 
  \hline
US>hh\_under 18&659.00&21.20\\ 
  US>hh\_over 65&857.00&27.60\\ 
  US>hh\_others&1592.00&51.20\\ 
   \hline
\end {tabular}
\end {table}

If you keep the original code (using jsarticle, you will use platex, so you will use

platex hoge.tex
dvipdfmxhoge.dvi

Compile like this.