I have a folder that that has a file added to it each day as below
Z:\old\stock110813.csv
Z:\old\stock120813.csv
Z:\old\stock130813.csv
Z:\old\stock140813.csv
I would like to import the latest file into SAS dynamically ie. searches the folder for the latest date
or would like sas to make a copy of the latest and change the name & location of the file
I have been searching the web for days testing little bits of code but am struggling therefore any help would be appreciated.
cheers John
If the date is predictable (ie, today's date) then you can do:
%let date=%sysfunc(today,dateformat.); *whatever dateformat you need;
proc import file="z:\old\stock&date..csv"... ;
run;
If not, then your best bet is to use a pipe with the directory listing to see the most recent filename. Something like this (directory logic depends on your server/OS/etc.)
filename mydir pipe 'dir /b /od /a-d z:\old\';
data myfiles;
infile mydir;
input #1 filename $32.;
call symput('filename',filename);
run;
proc import file="z:\old\&filename." ... ;
Do you want to use the system date or the date in the filename to determine what's the newest file? If you want to use the create of modify date, check out the foptname function to determine them. This code looks at the date in the filename. This code works without the need for X command authorization.
data newestFile (keep=newestFile);
format newestFile $20.;
retain newestFile newestDate;
rc = filename("dir","z:\old\");
did = dopen("dir");
/* loop through file and subdirectories in a directory */
do i = 1 to dnum(did);
csvFile = dread(did,i);
rc=filename("fid",cats("z:\old\",csvFile));
sdid=dopen("fid");
/*check if date in name is newest if it is a file */
if sdid le 0 then do;
csvFileDate = input(substr(csvFile,6,6),ddmmyy6.);
if csvFileDate gt newestDate then do;
newestDate = csvFileDate;
newestFile = csvFile;
end;
end;
else rc = dclose(sdid);
end;
rc = dclose(did);
/* move and rename file with latest date to newestFile.csv */
rc = rename(cats("z:\old\",newestFile), "z:\new\newestFile.csv",'file');
run;
Related
I have made a code of creating a folder that shall contain the output of the same code in a csv file. But when I wish to make amendments to the code so as to modify the output obtained in the csv file, I do not wish to run into FileExistsError. Is there any way I can do that? Sorry if the query is a foolish one, as I am just beginning to learn Python. Here's my code:
path = Path('file\location\DummyFolder')
path.mkdir(parents=True)
fpath = (path / 'example').with_suffix('.csv')
colours = ['red','blue', 'green', 'yellow']
colour_count = 0
with fpath.open(mode='w', newline='') as csvfile:
fieldnames = ['number', 'colour']
thewriter = csv.DictWriter(csvfile, fieldnames=fieldnames)
thewriter.writeheader()
for colour in colours:
colour_count+=1
thewriter.writerow({'number':colour_count, 'colour':colour})
In one of my folders (say Folder01) there are files like "IGN_A.txt", "IGN_B.txt", "IGN_C.txt".........
In another folder (say Folder02) there are files like "sim_IGN_A_M01.txt", "sim_IGN_A_M02.txt", "sim_IGN_A_M03.txt" for the corresponding file "IGN_A.txt" in Folder01.
Similarly, "sim_IGN_B_M01.txt", "sim_IGN_B_M02.txt", "sim_IGN_B_M03.txt" for corresponding file "IGN_B.txt" in Folder01.
How can I get the corresponding files from those Folders.
For example, I want to get "IGN_A.txt" along with "sim_IGN_A_M01.txt", "sim_IGN_A_M02.txt", "sim_IGN_A_M03.txt".
Here. I added my code which can only get "IGN_A.txt" along with "sim_IGN_A.txt".
Folder01 = 'Home/A1';
Folder02 = 'Home/A2';
%Going Throuh all the Folder01 files
Allfiles_Folder01 = dir(fullfile(Folder01, '*IGN*.txt'));
for k = 1:length(Allfiles_Folder01)
fullFileName = fullfile(Folder01, Allfiles_Folder01(k).name);
READ_Folder01=dlmread(fullFileName,'',2,0);
fullFileName_Sim = fullfile(Folder02, strcat('sim_',Allfiles_Folder01(k).name))
READ_Folder02=dlmread(fullFileName_Sim,'',1,0);
end
If the naming convention is consistent as provided by you, this would be my suggestion:
% Get all filenames from Folder01 in cell array.
Allfiles_Folder01 = dir(fullfile(Folder01, '*IGN*.txt'));
Allfiles_Folder01 = {Allfiles_Folder01.name}
% Iterate all filenames from Folder01.
for k = 1:numel(Allfiles_Folder01)
% Cut file extension from current filename.
filename = Allfiles_Folder01{k};
filename = filename(1:end-4);
% Get all filenames from Folder02 with specific search string in cell array.
Allfiles_Folder02 = dir(fullfile(Folder02, strcat('*', filename, '*.txt')));
Allfiles_Folder02 = {Allfiles_Folder02.name}
% Do stuff with filenames from Folder02 corresponding to filename from Folder01.
% ...
% ...
end
I have multiple folders (six or so) with multiple .CSV files in them. The CSV files are all of the same format:
Heading1,Heading2,Heading3
1,Monday,2.45
2,Monday,3.765...
Each .CSV has the same heading names [same data source for different months]. What is the best way to import these CSVs into SQL Server 2008? The server does not have xpShell configured [for security reasons which I cannot modify], so any method which uses that (which I originally tried), will not work.
EDIT
The CSV files are a maximum of 2mb in size and do not contain any commas (other than those required for delimiters).
Any ideas?
F.e. you got CSV file names sample.csv on D:\ drive, with this inside:
Heading1,Heading2,Heading3
1,Monday,2.45
2,Monday,3.765
Then you can use this query:
DECLARE #str nvarchar(max),
#x xml,
#head xml,
#sql nvarchar(max),
#params nvarchar(max) = '#x xml'
SELECT #str = BulkColumn
FROM OPENROWSET (BULK N'D:\sample.csv', SINGLE_CLOB) AS a
SELECT #head = CAST('<row><s>'+REPLACE(SUBSTRING(#str,1,CHARINDEX(CHAR(13)+CHAR(10),#str)-1),',','</s><s>')+'</s></row>' as xml)
SELECT #x = CAST('<row><s>'+REPLACE(REPLACE(SUBSTRING(#str,CHARINDEX(CHAR(10),#str)+1,LEN(#str)),CHAR(13)+CHAR(10),'</s></row><row><s>'),',','</s><s>')+'</s></row>' as xml)
SELECT #sql = N'
SELECT t.c.value(''s[1]'',''int'') '+QUOTENAME(t.c.value('s[1]','nvarchar(max)'))+',
t.c.value(''s[2]'',''nvarchar(max)'') '+QUOTENAME(t.c.value('s[2]','nvarchar(max)'))+',
t.c.value(''s[3]'',''decimal(15,7)'') '+QUOTENAME(t.c.value('s[3]','nvarchar(max)'))+'
FROM #x.nodes(''/row'') as t(c)'
FROM #head.nodes('/row') as t(c)
To get output like:
Heading1 Heading2 Heading3
1 Monday 2.4500000
2 Monday 3.7650000
At first we take data as SINGLE_CLOB with the help of OPEROWSET.
Then we put all in #str variable. The part from beginning to first \r\n we put in #head, the other part in #x with conversion to XML. Structure:
<row>
<s>Heading1</s>
<s>Heading2</s>
<s>Heading3</s>
</row>
<row>
<s>1</s>
<s>Monday</s>
<s>2.45</s>
</row>
<row>
<s>2</s>
<s>Monday</s>
<s>3.765</s>
</row>
After that we build dynamic query like:
SELECT t.c.value('s[1]','int') [Heading1],
t.c.value('s[2]','nvarchar(max)') [Heading2],
t.c.value('s[3]','decimal(15,7)') [Heading3]
FROM #x.nodes('/row') as t(c)
And execute it. Variable #x is passing as parameter.
Hope this helps you.
I ended up solving my problem using a non-SQL answer. Thank you everyone who helped contribute. I apologise for going with a completely off-field answer using PHP. Here is what I created to solve this problem:
<?php
//////////////////////////////////////////////////////////////////////////////////////////////////
// //
// Date: 21/10/2016. //
// Description: Insert CSV rows into pre-created SQL table with same column structure. //
// Notes: - PHP script needs server to execute. //
// - Can run line by line ('INSERT') or bulk ('BULK INSERT'). //
// - 'Bulk Insert' needs bulk insert user permissions. //
// //
// Currently only works under the following file structure: //
// | ROOT FOLDER //
// | FOLDER 1 //
// | CSV 1 //
// | CSV 2... //
// | FOLDER 2 //
// | CSV 1 //
// | CSV 2... //
// | FOLDER 3... //
// | CSV 1 //
// | CSV 2... //
// //
//////////////////////////////////////////////////////////////////////////////////////////////////
//Error log - must have folder pre-created to work
ini_set("error_log", "phplog/bulkinsertCSV.php.log");
//Set the name of the root directory here (Where the folder's of CSVs are)
$rootPath = '\\\networkserver\folder\rootfolderwithCSVs';
//Get an array with the folder names located at the root directory location
// The '0' is alphabetical ascending, '1' is descending.
$rootArray = scandir($rootPath, 0);
//Set Database Connection Details
$myServer = "SERVER";
$myUser = "USER";
$myPass = "PASSWORD";
$myDB = "DATABASE";
//Create connection to the database
$connection = odbc_connect("Driver={SQL Server};Server=$myServer;Database=$myDB;", $myUser, $myPass) or die("Couldn't connect to SQL Server on $myServer");
//Extend Database Connection timeout
set_time_limit(10000);
//Set to true for bulk insert, set to false for line by line insert
// [If set to TRUE] - MUST HAVE BULK INSERT PERMISSIONS TO WORK
$bulkinsert = true;
//For loop that goes through the folders and finds CSV files
loopThroughAllCSVs($rootArray, $rootPath);
//Once procedure finishes, close the connection
odbc_close($connection);
function loopThroughAllCSVs($folderArray, $root){
$fileFormat = '.csv';
for($x = 2; $x < sizeof($folderArray); $x++){
$eachFileinFolder = scandir($root."\\".$folderArray[$x]);
for($y = 0; $y < sizeof($eachFileinFolder); $y++){
$fullCSV_path = $root."\\".$folderArray[$x]."\\".$eachFileinFolder[$y];
if(substr_compare($fullCSV_path, $fileFormat, strlen($fullCSV_path)-strlen($fileFormat), strlen($fileFormat)) === 0){
parseCSV($fullCSV_path);
}
}
}
}
function parseCSV($path){
print_r($path);
print("<br>");
if($GLOBALS['bulkinsert'] === false){
$csv = array_map('str_getcsv', file($path));
array_shift($csv); //Remove Headers
foreach ($csv as $line){
writeLinetoDB($line);
}
}
else{
bulkInserttoDB($path);
}
}
function writeLinetoDB($line){
$tablename = "[DATABASE].[dbo].[TABLE]";
$insert = "INSERT INTO ".$tablename." (Column1,Column2,Column3,Column4,Column5,Column6,Column7)
VALUES ('".$line[0]."','".$line[1]."','".$line[2]."','".$line[3]."','".$line[4]."','".$line[5]."','".$line[6]."')";
$result = odbc_prepare($GLOBALS['connection'], $insert);
odbc_execute($result)or die(odbc_error($connection));
}
function bulkInserttoDB($csvPath){
$tablename = "[DATABASE].[dbo].[TABLE]";
$insert = "BULK
INSERT ".$tablename."
FROM '".$csvPath."'
WITH (FIELDTERMINATOR = ',', ROWTERMINATOR = '\\n')";
print_r($insert);
print_r("<br>");
$result = odbc_prepare($GLOBALS['connection'], $insert);
odbc_execute($result)or die(odbc_error($connection));
}
?>
I ended up using the script above to write to the database line by line... This was going to take hours. I modified to the script to use BULK INSERT which unfortunately we didn't have 'permissions' to use. Once I 'obtained' permissions, the BULK INSERT method worked a charm.
I have an operating JSON library which I use to load an array of tile IDs. When I double click main.lua directly from file explorer, it runs great, but when I open Corona Simulator and open my project from there or build my project and run it on my testing device, it gives me a null reference error when I attempt to use the data I loaded.
Here is the function to load a table from a JSON file:
function fileIO.loadJSONFile (fileName)
local path = fileName
local contents = ""
local loadingTable = {}
local file = io.open (path, "r")
print (file)
if file then
local contents = file:read ("*a")
loadingTable = json.decode (contents)
io.close (file)
return loadingTable
end
return nil
end
Here is the usage:
function wr:renderChunkFile (path)
local data = fileIO.loadJSONFile (path)
self:renderChunk (data)
end
function wr:renderChunk (data)
local a, b = 1
if (self.img ~= nil) then
a = #self.img + 1
self.img[a] = {}
else
self.img[1] = {}
end
if (self.chunks ~= nil) then
b = #self.chunks + 1
self.chunks[b] = display.newGroup ()
else
self.chunks[1] = display.newGroup ()
end
for i = 1, #data do -- Y axis ERROR IS HERE
self.img[a][i] = {}
for j = 1, #data[i] do -- Z axis
self.img[a][i][j] = {}
for k = 1, #data[i][j] do -- X axis
if (data[i + 1] ~= nil) then
if (data[i + 1][j][k] < self.transparentLimit) then
self.img[a][i][j][k] = display.newImage ("images/tiles/"..data[i][j][k]..".png", k*self.tileWidth, display.contentHeight -j*self.tileDepth - i*self.tileThickness)
self.chunks[b]:insert (self.img[a][i][j][k])
elseif(data[i + 1] == nil) then
self.img[a][i][j][k] = display.newImage ("images/tiles/"..data[i][j][k]..".png", k*self.tileWidth, display.contentHeight -j*self.tileDepth - i*self.tileThickness)
self.chunks[b]:insert (self.img[a][i][j][k])
end
end
end
end
end
end
When it gets to the line for i = 1, #data do it tells me it is trying to access the length of a nil field. Where did I go wrong here?
EDIT: I feel the need to give a more clear explanation of what my problem is. I am getting inconsistent results from this program. When I select main.lua in file explorer and open it with Corona Simulator, it works. When I open Corona Simulator and internally navigate to main.lua, it does not work. When I build the project and test it on my device, it does not work. What I really need is some insight into Corona's JSON library and APK internal directory structure requirements (directory nesting limits, naming restrictions, etc.). If someone thinks of something else that might cause the issue I am having, please bring it up! I am open to anything.
Without seeing the entire error message and not knowing what the value of "path" is it's going to be hard to speculate. But Corona SDK uses four base directories:
system.ResourceDirectory -- Same folder as main.lua and is read-only
system.DocumentsDirectory -- Your writable folder where your data lives
system.CachesDirectory -- for downloaded files
system.TemporaryDirectory -- for temp files.
The last three, while in the simulator are in the project's Sandbox master folder. On device who knows where the folders really are.
In your case, if your JSON file is going to be included in with your downloadble app, your .json file should be in the same folder with your main.lua (or a sub folder) and referenced in system.ResourceDirectory.
I currently have a MATLAB function that looks like this:
function outfile=multi_read(modelfrom,modelto,type)
models=[modelfrom:1:modelto];
num_models=length(models);
model_path='../MODELS/GRADIENT/'
for id=1:num_models
fn=[model_path num2str(models(id)) '/']; %Location of file to be read
outfile=model_read(fn,type); %model_read is a separate function
end
end
The idea of this function is to execute another function model_read for a series of files, and output these files to the workspace (not to disk). Note that the output from model_read is a structure! I want the function to save the file to the workspace using sequential names, similar to typing:
file1=multi_read(1,1,x)
file2=multi_read(2,2,x)
file3=multi_read(3,3,x)
etc.
which would give file1, file2 and file3 in the workspace, but instead by recalling the command only once, something like:
multi_read(1,3,x)
which would give the same workspace output.
Essentially my questions is, how do I get a function to output variables with multiple names without having to recall the function multiple times.
As suggested in the comment I would try this approach which is more robust, at least IMHO:
N = tot_num_of_your_files; %whatever it is
file = cellfun(#(i)multi_read(i,i,x),mat2cell(1:N,1,ones(1,N)),...
'UniformOutput' , false); %(x needs to be defined)
You will recover objects by doing file{i}.
Here is code to do what you ask:
for i = 1:3
istr=num2str(i)
line = ['file' istr '= multi_read(' istr ', ' istr ', x)']
eval(line)
end
Alternatively, here is code to do what you should want:
for i = 1:3
file{i} = multi_read(i,i,x)
end