I have a for loop iterating over a folder of one column csv's using glob, it makes some adjustments and then appends the results to a list and saves to a new csv, it resembles:
data= []
infiles = glob.glob("*.csv")
for file in infiles:
df = pd.io.parsers.read_csv(file)
(assorted adjustments)
data.append(df)
fullpanel = pd.concat(panel)
fullpanel.to_csv('data.csv')
The problem is that makes one long column, I need each column (of differing lengths) added next to each other.
I think you can add parameter axis=1 to concat for columns added next to each other. Also you can change pd.io.parsers.read_csv to pd.read_csv and panel to data in concat.
data= []
infiles = glob.glob("*.csv")
for file in infiles:
df = pd.read_csv(file)
(assorted adjustments)
data.append(df)
fullpanel = pd.concat(data, axis=1)
fullpanel.to_csv('data.csv')
Related
I have to filter the data, therefore I need to create new CSV file based on the filters.
I am having a trouble doing it, cause the new file does not change after I run the code
Below is my code. Where I have two csv file. Stage_3_try.csv file is the one I am trying to add new data. I used enumerate to get the index value of the specific value I searched in previous csv file.
# Projec
import csv
from csv import writer
A = np.array([ 316143.8829, 6188926.04])
B = np.array([ 314288.7418, 6190277.519])
for i in range(0,len(east_3)):
P = []
P.append(east_3[i])
P.append( north_3[i])
P = np.asarray(P)
projected = point_on_line(P) #a code to do the projection
x_values = [A[0], B[0]]
y_values = [A[1], B[1]]
plt.plot(x_values, y_values, 'b-')
if projected[0]>315745.75 and projected[1]>6188289:
with open('Stage_3_try.csv', 'a') as f_out:
writer = csv.writer(f_out)
for num, row in enumerate(stage_3['UTM North NAD83']):
if row == P[1]:
writer.writerow(stage_3.loc[[num][0]])
print(type(stage_3.loc[[num][0]]))
plt.plot(projected[0], projected[1], 'rx')
f_out.close()
else:
pass
PS: I updated the code, since the previous one worked, but when I added it to the loop, it stopped working
I am trying to grab data from a mysql database and put it in an excel template ( with macro's).
The template has mutiple sheets.I want to put the data in a specific sheet and specific cell ( B2 ) since the sheet already contains data.
The code i am using is:
wb= openpyxl.load_workbook('C:/Users/Olav/Desktop/Xenos/Nieuw.xlsx')
ws = wb['Dump Pickloc - del. web']
picklocaties = "SELECT Artikelnummer, Locatie,PICKZONE FROM picklocaties WHERE PICKZONE in ('BASIS','HL')"
df = pd.read_sql(sql=picklocaties, con=mydb)
rows = dataframe_to_rows(df)
for r in dataframe_to_rows(df, index=False, header=False):
ws.append(r)
I tryed using to_excel but that just deletes everything.
The template in which i am putting the data looks like This.
It would be great if this code would work but it does not have that option:
for r in dataframe_to_rows(df, index=False, header=False, startrow=1, startcol=1):
ws.append(r) \
Woah i'm half way there, woah living on prayer.
This codes gets me halfway. I get the columns now where i want without messing up the rest. But for some reason the rest of the data is not shown.
for col, text in enumerate(df, start=2):
ws.cell(column=col,row=2, value=text)
I have a CSV file where 2 columns contain several different text values e.g.
Column 1: Reptiles, Health, Hygiene
Column 2: Purity
I need to use VBscript to split these columns into a new CSV file without changing the current file, expected output in new CSV file shown below:
Column 1 Column 2
Reptiles Reptiles
Health Health
Hygiene Hygiene
Purity Purity
Unfortunately(?) it must be done with VB Script and nothing else.
Here is an example of how the data looks (of course the data consistently repeats with some extra entries through the same columns in file 1.
And here is an example of how it needs to look but it needs to repeat down until all unique entries from Column 1 and 2 in the original file have been input as a single entry to Column 1 in the new file and copied to Column 2 in the same new file. e.g.
Examples in text format as requested:
Original file:
Column 1,Column 2
"Reptiles, Health, Hygiene",Purity
New File:
Column 1,Column 2
Reptiles,Reptiles
Health,Health
Hygiene,Hygiene
Purity,Purity
I think this is a simple matter of using the FileSystemObject with Split function.
Assuming each input line is just one set of data you can remove the double quotes and process from there
Try this VB script out (edited to process header line separately):
Const Overwrite = True
Set ObjFso = CreateObject("Scripting.FileSystemObject")
Set ObjOutFile = ObjFso.CreateTextFile("My New File Path", Overwrite)
Set ObjInFile = ObjFso.OpenTextFile("My Old File Path")
' Skip processing first header line and just write it out as is
strLine = ObjInFile.ReadLine
ObjOutFile.WriteLine strLine
Do Until ObjInFile.AtEndOfStream
' Remove all double quotes to treat this as one set of data
strLine = Replace(ObjInFile.ReadLine, """","")
varData = Split(strLine,",")
' Write out each element twice into its own line
For i = 0 to uBound(varData)
ObjOutFile.WriteLine varData(i) & "," & varData(i)
Next i
Loop
ObjInFile.Close
ObjOutFile.Close
I am working with a text file and need to call out the sum found from my last column of data [4] that I have made. I have done everything I need for the last column and have used total += square to add the first value in row one with the next value in row two and so on till I hit my 100th row in my text file. Now I need to be able to take my sum that I want in my 100 row and store it as a variable. How can I go about calling it out?
fullPath = open("localzscoretest.txt", "r") #Where I have our the current table located
import math
def globalchiSquare(fullPath):
for line in fullPath:
line = line.strip() #Strip it
lines = line.split(',') #split it
rows = lines[1:] #keeping the numbers
rows = map(float, rows) #getting my numbers in the .txt ready for the equation
square = (rows[4]**2) #squared the z score column
total += square
print total
globalchiSquare(fullPath)
change
square = (rows[4]**2) #squared the z score column
to be
square += (rows[4]**2) #squared the z score column
Give globalchiSquare a readlines() method in order to iterate.
In the function do
def globalchiSquare(fullPath):
for line in fullPath.readlines():
. . .
You should also keep your variables clear. When you say lines, it seems like you are saying that there are multiple--rows, too.
Just make it more simple and include the sum.
def globalchiSquare(fullPath):
total = 0
for line in fullPath.readlines(): # readlines() method
line = line.strip() # cut off ends
line = line.split(',') # create list
row = line[1:] # create row from line
row = map(float,row) # convert to floats
square = row[4]**2 # find square
print 'square',square
total += square
print 'total',total
return total
my_var = globalchiSquare(fullPath)
print my_var # should give total
EDIT: The return statement allows you to store the value of total.
I've read through the Highcharts how-to, checked the demo galleries, searched google, read the X amount of exact similar threads here on stackoverflow yet I cannot get it to work.
I'm logging data in a csv file in the form of date,value.
Here's what the date looks like
1355417598678,22.25
1355417620144,22.25
1355417625616,22.312
1355417630851,22.375
1355417633906,22.437
1355417637134,22.437
1355417641239,22.5
1355417641775,22.562
1355417662373,22.125
1355417704368,21.625
And this is how far I've managed to get the code:
http://jsfiddle.net/whz7P/
This renders a chart, but with no series or data at all. I think I'm fudging things up while formatting the data so it can be interpreted in highcharts.
Anyone able to give a helping hand?
So, you have the following data structure, right ?
1355417598678,22.25
1355417620144,22.25
1355417625616,22.312
1355417630851,22.375
1355417633906,22.437
1355417637134,22.437
1355417641239,22.5
1355417641775,22.562
1355417662373,22.125
1355417704368,21.625
Then you split it into an array of lines, so each array item is a line.
Then for each line you do the following.
var items = line.split(';'); // wrong, use ','
But there ins't ; into the line, you should split using ,.
The result will be a multidimencional array which each item is an array with the following structure. It will be stored in a var named data.
"1355417598678","22.25" // date in utc, value
This is the expected data for each serie, so you can pass it directly to your serie.
var serie = {
data: data,
name: 'serie1' // chose a name
}
The result will be a working chart.
So everything can be resumed to the following.
var lines = data.split('\n');
lines = lines.map(function(line) {
var data = line.split(',');
data[1] = parseFloat(data[1]);
return data;
});
var series = {
data: lines,
name: 'serie1'
};
options.series.push(series);
Looking at your line.split part:
$.get('data.csv', function(data) {
// Split the lines
var lines = data.split('\n');
$.each(lines, function(lineNo, line) {
var items = line.split(';');
It looks like you are trying to split on a semi-colon (;) instead of a comma (,) which is what is in your sample CSV data.
You need to put
$(document).ready(function() {
in the 1st line, and
});
in the last line of the javascript to make this work.
Could you upload your csv file? Is it identical to what you wrote in your original post? I ran into the same problem, and it turns out there are errors in the data file.