I want to register a lot of data with bulkAPI in elasticsearch with the in_http plug-in of fluentd.

Asked 1 years ago, Updated 1 years ago, 73 views

Use the in_http plug-in in Fluentd to
I would like to send the json file and register it with elasticsearch.

Currently, I am trying to handle the following json files:

{
    "user_id": 0,
    "time"—0,
    "data": [
        {
            "data0": 0.0,
            "data1"—0.1
        },
        {
            "data0": 0.0,
            "data1"—0.2
        }
    ]
}

I was able to confirm that one piece of data was successfully registered.

"However, I would like to register each element of the ""data"" array as a different id, so
" To make bulkAPI available,

{"index":{"_index":<index_name>, "_type":<type_name>, "_id":<id>}}
{ "user_id":0, "time":0, "data":{"data0":0.0, "data1":0.1}
{ "index": {"_index": <index_name>, "_type": <type_name>, "_id": <id>}
{ "user_id":0, "time":0, "data":{"data0":0.0, "data1":0.2}}

We are considering converting it to the form shown in and throwing it into elasticsearch.

About its implementation:
·How to pass index_name/type_name to Fluentd via http POST
·How to convert to bulkAPI format on fluidd
I'm in trouble because I don't know.

For instructions on registering in bulkAPI format with fluidd-elasticsearch, see
If anyone knows a good way, please let me know.
Thank you.
 
 
The various versions are as follows.
OS:CentOS Linux release 7.5.1804(Core)
td-agent:2.3.6
elasticsearch:5.5.3

elasticsearch fluentd

2022-09-30 21:32

1 Answers

Use fluent-plguin-elasticsearch.

In_http is used differently because it allows Fluentd to accept HTTP requests.


2022-09-30 21:32

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.