How pandas to parse CSV file containing many data sets format TimeStamp0; Value0; Timestamp1;Value1;?

Hello.

I just started learning Python and decided to try to solve the urgent practical task of creating interactive reports.

Sketched the module that is specific to a CSV file. The result was, but I want to do well.

The next step is unification to import a file with an unknown number of datasets. And here then the question arose about how best to process the source data.

Data in CSV is in the format:

Timestamp0; Value0 ; Timestamp1; Value1; ...TimestampN; ValueN;

The main problem in understanding the organization of the index. When reading means Pandas you can select a column or series as the index. And my problem is that for each column of data must have its own index. For further charting tools Plotly simply repeatedly read CSV required column in index_col. Understand that this is very wrong, but can't figure out how to do correctly.

Could You tell me how to work with such data?
The current implementation
import plotly
import plotly.graph_objs as go
#from plotly.graph_objs import Scatter, Layout

import pandas

plotly.offline.init_notebook_mode(connected=True) #initialize a plotly work offline
# read CSV file

source="d:/TEMP/GAS.csv"
trend1 = 'BK1 Thermocouple, °C'
trend2 = 'Thermocouple BK2, °C'
trend3 = 'Thermocouple BK3, °C'
trend1_name='A-A. BK1'
trend2_name='A-A. BK2'
trend3_name='A-A. BK3'

df = pandas.read_csv(source,
sep=';',
 parse_dates=[trend1 + 'Time'],
dayfirst=True,
 index_col=[trend1 + 'Time'],
decimal=','
)


dfBK2 = pandas.read_csv(source,
sep=';',
 parse_dates=[trend2 + 'Time'],
dayfirst=True,
 index_col=[trend2 +' Time'],
decimal=','
)

dfBK3 = pandas.read_csv(source,
sep=';',
 parse_dates=[trend3 + 'Time'],
dayfirst=True,
 index_col=[trend3 +' Time'],
decimal=','
)

# define trends
trace1 = go.Scatter(x=df.index
 y=df[trend1 + 'ValueY'],
name=trend1_name
)

trace2 = go.Scatter(x=dfBK2.index
 y=dfBK2[trend2 + 'ValueY'],
name=trend2_name,
yaxis='y2'
)
trace3 = go.Scatter(x=dfBK3.index
 y=dfBK3[trend3 + 'ValueY'],
name=trend3_name,
yaxis='y3'
)
data = [trace1, trace2, trace3]

# definition of the plot area
Width = 1
High = 1
domainWidth=Width-0.1
layout = dict(legend= dict(x= 0, 
 y= 1
),
hovermode='x'
 xaxis=dict(domain=[0, domainWidth] # size of the graph area 
 ), 
 yaxis=dict(showgrid=True, 
 side= 'right', 
 title= trend1_name
 ), 

 yaxis2=dict(overlaying= 'y', 
 anchor= 'free' 
 side= 'right',
 title= trend2_name,
 position=domainWidth+0.05 
),
 yaxis3=dict(overlaying= 'y', 
 anchor= 'free' 
 side= 'right',
 title= trend3_name,
 position=domainWidth+0.1 
),
)
fig = dict(data=data,layout=layout)
plotly.offline.plot(fig)
Link to an example file, if anyone is interested
June 10th 19 at 16:56
1 answer
June 10th 19 at 16:58
Solution
You can simply split the table into several independent dataframes. To read the CSV multiple times is not necessary.

For example:

import pandas as pd
import numpy as np


def get_df(filename, parse_dates=None):
 """Build :class:`DataFrame` list from CSV file.

 Expected CSV file format::

 Timestamp0; Value0 ; Timestamp1; Value1; ...TimestampN; ValueN;

Args:
 filename: CSV filename.
 parse_dates: List of columns with dates.

Returns:
 List of DataFrames with 'Timestamp' as index and 'Value' as value
column.

Notes:
 :attr:`DataFrame._name` contains the name extracted
 from 'TimestampX' column.
"""

 df_all = pd.read_csv(filename, sep=';', decimal=',',
 parse_dates=parse_dates, header=0)

 assert len(df_all.columns) % 2 == 0

 lst = []
 columns = ['time', 'value']

 # Create lsit of 2-items chunks.
 col_list = np.split(df_all.columns, len(df_all.columns) / 2)

 for cols in col_list:
 df = df_all[cols] # split a 2-column DataFrame.
 df._name = cols[0].split(',')[0] # attach data frame name to.
 df.columns = columns # change columns names.
 df = df.set_index('time') # set index to timestamps.
lst.append(df)

 return lst


df_list = get_df('GAS.csv', parse_dates=[0, 2, 4])
df = df_list[0]
print(df.index)
print(df._name)

...
Sergey, thank you very much for the help! - nikita.Stracke commented on June 10th 19 at 17:01
Not for that. - Marcus64 commented on June 10th 19 at 17:04

Find more questions by tags CSVPython