AWS cloudwatch logs cannot be molded.

Run Environment
OS:windows10vscode
languages:python3
libraries:standard libraries only (can't use pandas, numpy, etc.)

About the program you are creating
I am creating a program to mold the AWS CloudWatch Logs log from csv.
When you retrieve csv data from CloudWatch, you get the following csv data:

Cl Some csv data for CloudWatch logs


timestamp, message
000000000000, "START RequestId: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
"
111111111, "[INFO] 2022-12-09T00:45:15.119ZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
"
2222222222222, "[INFO] 2022-12-09T00:45:15.120ZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
"

If you look at this log, you will find the log level date time, RequestId, and log together.
In this condition, if you attach them to Excel, they will stick together and reflect them.
Therefore, in order to prevent these data from sticking together, we are thinking that the following csv data is for molding.

成形 After molding csv data

timestamp, message
000000000000,START RequestId:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
111111111, [INFO], 2022-12-09T00:45:15.119Z, AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
2222222222222, [INFO], 2022-12-09T00:45:15.120Z, AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

やりたいWhat I want to do
I want to mold csv data as above, but it doesn't work because it's lined up vertically.
I would appreciate it if you could tell me how to do it.

試What I tried
As shown in the code below:
①Read the actual log and create an array of csvalldata.
②As the log level, date, etc. are attached to the first part of the array, split by \t.
　(When I listed the csv file, the log level and date spaces were displayed as \t.)
③Add the 0th corresponding col_0 to the split list splitlist.
④Add splitlist to csvalldata.Loop data
⑤If you csvize the list as it is, [ ] will be included, so take the elements out of the list and rearrange them.

I thought about it in my own way and wrote it, but I couldn't get the csv data.
I would appreciate it if you could tell me the cause.

Also, I would appreciate it if you could let me know if there is a more efficient way.

import csv
import datetime

# Load open() file
file=open("log_before.csv", "r", encoding="utf-8") 

# Loading the csv File
reader=csv.reader(file)

csvalldata=[ ]
for num in reader:
    col_0 = num[0] 
    col_1 = num[1]
    splitlist=col_1.split('\t')
    splitlist.insert(0,col_0)# Add 0th element
    csvalldata.append(splitlist)

print("A. Check List Status", csvalldata)          
newdata='"   
for i in csvalldata:
  for jini:
    newdata=newdata+'+j+'','
    print("newdata", newdata)

# Specify the read encoding for the open()
now=datetime.datetime.now()#Getting the current time
out_filename = 'log_after_{0:%M%S}.csv'.format(now) 
newfile=open(out_filename, "w", encoding="utf-8") 

# Writing a csv file
writer=csv.writer(newfile)
writer.writerows (newdata)
print("Create csv file:", out_filename)

python aws csv excel

2022-12-11 15:55

1 Answers

libraries:standard libraries only (can't use pandas, numpy, etc.)

Also, I would appreciate it if you could let me know if there is a more efficient way.

In a library-constrained environment, the code is likely to be as muddy as the VBA.If you're going to eventually load it into Excel, I recommend that you process it in Excel from the beginning.Excel has a feature called Power Query for data processing like CSV file shaping.
Choose Data - New Query - Empty Query to launch the Power Query Editor.Select "Advanced Editor" here to enter code.

let
    csv=Csv.Document(File.Contents("C:\Path\To\log_before.csv"), [Delimiter=", ", Encoding=65001, QuoteStyle=QuoteStyle.Csv],
    headered=Table.PromoteHeaders(csv, [PromoteAllScalars=true]),
    split=Table.SplitColumn(headed, "message", Splitter.SplitTextByDelimiter("#(tab), QuoteStyle.None), {"col_0", "col_1", "col_2", "col_3" },
    transformed=Table.TransformColumnTypes(splitted, {{"timestamp", Int64.Type}, {"col_0", type text}, {"col_1", type datetime}, {"col_2", type text}, {"col_3", type text}})
in
    transformed

Excel maintains how it was loaded, so you can try again from Refresh menu.You can use Folder.Files to search for files even if the file name changes dynamically.

2022-12-11 18:12

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656