I am using Paraview to display well log information stored in csv files.
Upon loading the csv data file and adding the "Table to Points" filter, I manually specify which columns should be used for the X Column, Y Column and Z Column (e.g. I am using a column called "Easting" for the "X Column").
Is there a way to automate this step? Maybe some kind of configuration file that could be fed into Paraview to indicate the column mapping?
You can use python macro to do that. Macros in ParaView are python script you can trigger from the toolbar.
You can use this one to create and configure a TableToPoints filter (with modif according to your column names):
#### import the simple module from the paraview
from paraview.simple import *
#### disable automatic camera reset on 'Show'
paraview.simple._DisableFirstRenderCameraReset()
# create a new 'Table To Points'
tableToPoints1 = TableToPoints(registrationName='TableToPoints1')
# configure column names
tableToPoints1.XColumn = 'y1'
tableToPoints1.YColumn = 'y2'
tableToPoints1.ZColumn = 'y26'
# get active view
spreadSheetView1 = GetActiveViewOrCreate('SpreadSheetView')
# show data in view
tableToPoints1Display = Show(tableToPoints1, spreadSheetView1, 'SpreadSheetRepresentation')
# hide data in view
Hide(sineWavescsv, spreadSheetView1)
# update the view to ensure updated data information
spreadSheetView1.Update()
Store it on your disk and go to menu Macros / Import new macro
Note that you can easily create your own with the Python Trace:
menu Tools / Start Trace (with default options)
perform the actions you want to replay later (load file, create filters, edit color map, ...)
Tools / Stop Trace. It opens an editor with python code, you can save it as a macro.
The macro is a new button in the toolbar.
Related
I have a dataset which have an xml column and i am trying to export individual xmls as files with filename being in another column using codeworkbook
I filtered the rows i want using below code
def prepare_input(xml_with_debug):
from pyspark.sql import functions as F
filter_column = "key"
filter_value = "test_key"
df_filtered = xml_with_debug.filter(filter_value == F.col(filter_column))
approx_number_of_rows = 1
sample_percent = float(approx_number_of_rows) / df_filtered.count()
df_sampled = df_filtered.sample(False, sample_percent, seed=0)
important_columns = ["key", "xml"]
return df_sampled.select([F.col(c).cast(F.StringType()).alias(c) for c in important_columns])
It works till here. Now for the last part i tried this in a python task, but was complaining about the parameters (i should have set it up wrongly). But even if it works it will be as a single file i think.
from transforms.api import transform, Input, Output
#transform(
output=Output("/path/to/python_csv"),
my_input=Input("/path/to/input")
)
def my_compute_function(output, my_input):
output.write_dataframe(my_input.dataframe().coalesce(1), output_format="csv", options={"header": "true"})
I am trying to set it up in GUI like below
My question i guess is, what will be the code in the last Python task (write_file) after the prepare input so that i extract individual xmls (And if possible zip them into single file for download)
You can access the output dataset filesystem and write files into it in whatever format you want.
The documentation for that can be found here: https://www.palantir.com/docs/foundry/code-workbook/transforms-unstructured/#writing-files
(If you want to do it from a code repository it's very similar https://www.palantir.com/docs/foundry/transforms-python/unstructured-files/#writing-files)
By doing that you can create multiple different files or you can create a single zip file and write it into a dataset.
I have a text widget in which user needs to feed in batch id say "201906", it is a year with month. So the data gets processed for this particular batch. So now how do I get this value from a CSV or a file name located in ADLS container and use it in a databricks dropdown widget so that the user will not have the freedom to enter a batchid which is not to be processed or restricted to process? So basically I want to give the option to the user with the required batch to be processed but not an entire field to type whatever he wants.
It's just easy - you can use local file API to access file on DBFS, like this (you need to replace dbfs:/ with /dbfs/ to access file on DBFS):
with open("/dbfs/tmp/my-batches.txt") as f:
batches = [l.strip() for l in f.readlines() if l.strip() != ""]
dbutils.widgets.dropdown(name="batches", label="Select batch",
choices=batches, defaultValue=batches[0])
will give you what you need:
You can achieve the same by using the Spark API - it could be a bit slower, but it won't require that storage account is mounted - you can use abfss://, wasbs:// and other supported protocols:
dbutils.widgets.removeAll()
df = spark.read.text("/tmp/my-batches.txt")
batches = [r[0].strip() for r in df.collect() if r[0].strip() != ""]
dbutils.widgets.dropdown(name="batches", label="Select batch",
choices=batches, defaultValue=batches[0])
When I run my PsychoPy experiment, PsychoPy saves a CSV file that contains my trials and the values of my variables.
Among these, there are some variables I would like to NOT be included. There are some variables which I decided to include in the CSV, but many others which automatically felt in it.
is there a way to manually force (from the code block) the exclusion of some variables in the CSV?
is there a way to decide the order of the saved columns/variables in the CSV?
It is not really important and I know I could just create myself an output file without using the one of PsychoPy, or I can easily clean it afterwards but I was just curious.
PsychoPy spits out all the variables it thinks you could need. If you want to drop some of them, that is a task for the analysis stage, and is easily done in any processing pipeline. Unless you are analysing data in a spreadsheet (which you really shouldn't), the number of columns in the output file shouldn't really be an issue. The philosophy is that you shouldn't back yourself into a corner by discarding data at the recording stage - what about the reviewer who asks about the influence of a variable that you didn't think was important?
If you are using the Builder interface, the saving of onset & offset times for each component is optional, and is controlled in the "data" tab of each component dialog.
The order of variables is also not under direct control of the user, but again, can be easily manipulated at the analysis stage.
As you note, you can of course write code to save custom output files of your own design.
there is a special block called session_variable_order: [var1, var2, var3] in experiment_config.yaml file, which you probably should be using; also, you should consider these methods:
from psychopy import data
data.ExperimentHandler.saveAsWideText(fileName = 'exp_handler.csv', delim='\t', sortColumns = False, encoding = 'utf-8')
data.TrialHandler.saveAsText(fileName = 'trial_handler.txt', delim=',', encoding = 'utf-8', dataOut = ('n', 'all_mean', 'all_raw'), summarised = False)
notice the sortColumns and dataOut params
I am building a program on Animate CC 2019 which needs to access a text file where it can read from and write to.
I want to store strings and numbers, from Input Texts, then write them in the file, in addition to the already existent contents.
I also need to be able to manage how the data is written in the file. So, for example, I want to have 2 arrays: Array1 stores names and Array2 stores numbers. So when the program writes to the file, it needs to write on each line the corresponding values of a specific index in the arrays. So if Array1(0) = name1 & Array2(0) = 52, in the file it should be written:
name1 52
If not exactly like this, a way which data can be written in a manageable way.
I am trying this on Windows 10/AIR.
My questions:
How can I manage how data are written in a file?
Which code should I use? I find info on two ways to approach this:
a. Code using “File” and “FileStream” which does NOT work! I get error messages: “Type was not found or was not a compile-time constant” for both, like the system does not recognize the commands. Are these commands still usable on the current version of Animate CC?
b. The code snipets/AIR code which does work, however, I cannot find a way to manage how data are written in the file, as explained above. Everytime something is written in the file, the previous data are overwritten.
Below the codesnipets code which works.
/* Click to Write to a Text File
Clicking on the specified object will prompt the user to select a location and file name and save a text string to it.
Instructions:
To set what data is saved to the file change "Text to save to file." to the data you wish to save.
*/
import flash.filesystem.File;
import flash.filesystem.FileMode;
import flash.filesystem.FileStream;
movieClip_1.addEventListener(MouseEvent.CLICK, fl_SaveFile);
var fl_FileDataToSave:String = "Text to save to file.";
var fl_SaveFileStream:FileStream = new FileStream(); // FileStream used to write to the file
var fl_SaveFileChooser:File = File.documentsDirectory; // Default to the documents directory
fl_SaveFileChooser.addEventListener(Event.SELECT, fl_WriteFileHandler);
// Opens a Save As dialog box for user to specify a file name
function fl_SaveFile(event:MouseEvent):void
{
fl_SaveFileChooser.browseForSave("Save As:");
}
// Write data in fl_FileDataToSave variable
function fl_WriteFileHandler(event:Event):void
{
fl_SaveFileChooser = event.target as File;
fl_SaveFileStream = new FileStream();
fl_SaveFileStream.openAsync(fl_SaveFileChooser, FileMode.WRITE);
fl_SaveFileStream.writeMultiByte(fl_FileDataToSave, File.systemCharset);
fl_SaveFileStream.close();
}
As of now I use 3 Notebook :
Functions
Where I have all the functions I created and call in the other Notebooks.
Transformation
Based on the original data, I compute transformations and add columns/List
When data is my raw data, I then call :
t1data : the result of the first transformation
t2data : the result of the second transformation
and so on,
I am yet at t20.
Display & Analysis
Using both the above I create Manipulate object that enable me to analyze the data.
Questions
Is there away to save the results of the Transformation Notebook such that t13data for example can be used in the Display & Analysis Notebooks without running all the previous computations (t1,t2,t3...t12) it is based on ?
Is there a way to use my Functions or transformed data without opening the corresponding Notebook ?
Does my separation strategy make sense at all ?
As of now I systematically open the 3 and have to run them all before being able to do anything, and it takes a while given my poor computing power and yet inefficient codes.
Saving variable states: can be done using DumpSave, Save or Put. Read back using Get or <<
You could make a package from your functions and read those back using Needs or <<
It's not something I usually do. I opt for a monolithic notebook containing everything (nicely layered with sections and subsections so that you can fold open or close) or for a package + slightly leaner analysis notebook depending on the weather and some other hidden variables.
Saving intermediate results
The native file format for Mathematica expressions is the .m file. This is human readable text format, and you can view the file in a text editor if you ever doubt what is, or is not being saved. You can load these files using Get. The shorthand form for Get is:
<< "filename.m"
Using Get will replace or refresh any existing assignments that are explicitly made in the .m file.
Saving intermediate results that are simple assignments (dat = ...) may be done with Put. The shorthand form for Put is:
dat >> "dat.m"
This saves only the assigned expression itself; to restore the definition you must use:
dat = << "dat.m"
See also PutAppend for appending data to a .m file as new results are created.
Saving results and function definitions that are complex assignments is done with Save. Examples of such assignments include:
f[x_] := subfunc[x, 2]
g[1] = "cat"
g[2] = "dog"
nCr = #!/(#2! (# - #2)!) &;
nPr = nCr[##] #2! &;
For the last example, the complexity is that nPr depends on nCr. Using Save it is sufficient to save only nPr to get a fully working definition of nPr: the definition of nCr will automatically be saved as well. The syntax is:
Save["nPr.m", nPr]
Using Save the assignments themselves are saved; to restore the definitions use:
<< "nPr.m" ;
Moving functions to a Package
In addition to Put and Save, or manual creation in a text editor, .m files may be generated automatically. This is done by creating a Notebook and setting Cell > Cell Properties > Initialization Cell on the cells that contain your function definitions. When you save the Notebook for the first time, Mathematica will ask if you want to create an Auto Save Package. Do so, and Mathematica will generate a .m file in parallel to the .nb file, containing the contents of all Initialization Cells in the Notebook. Further, it will update this .m file every time you save the Notebook, so you never need to manually update it.
Sine all Initialization Cells will be saved to the parallel .m file, I recommend using the Notebook only for the generation of this Package, and not also for the rest of your computations.
When managing functions, one must consider context. Not all functions should be global at all times. A series of related functions should often be kept in its own context which can then be easily exposed to or removed from $ContextPath. Further, a series of functions often rely on subfunctions that do not need to be called outside of the primary functions, therefore these subfunctions should not be global. All of this relates to Package creation. Incidentally, it also relates to the formatting of code, because knowing that not all subfunctions must be exposed as global gives one the freedom to move many subfunctions to the "top level" of the code, that is, outside of Module or other scoping constructs, without conflicting with global symbols.
Package creation is a complex topic. You should familiarize yourself with Begin, BeginPackage, End and EndPackage to better understand it, but here is a simple framework to get you started. You can follow it as a template for the time being.
This is an old definition I used before DeleteDuplicates existed:
BeginPackage["UU`"]
UnsortedUnion::usage = "UnsortedUnion works like Union, but doesn't \
return a sorted list. \nThis function is considerably slower than \
Union though."
Begin["`Private`"]
UnsortedUnion =
Module[{f}, f[y_] := (f[y] = Sequence[]; y); f /# Join###] &
End[]
EndPackage[]
Everything above goes in Initialization Cells. You can insert Text cells, Sections, or even other input cells without harming the generated Package: only the contents of the Initialization Cells will be exported.
BeginPackage defines the Context that your functions will belong to, and disables all non-System` definitions, preventing collisions. (There are ways to call other functions from your package, but that is better for another question).
By convention, a ::usage message is defined for each function that it to be accessible outside the package itself. This is not superfluous! While there are other methods, without this, you will not expose your function in the visible Context.
Next, you Begin a context that is for the package alone, conventionally "`Private`". After this point any symbols you define (that are not used outside of this Begin/End block) will not be exposed globally after the Package is loaded, and will therefore not collide with Global` symbols.
After your function definition(s), you close the block with End[]. You may use as many Begin/End blocks as you like, and I typically use a separate one for each function, though it is not required.
Finally, close with EndPackage[] to restore the environment to what it was before using BeginPackage.
After you save the Notebook and generate the .m package (let's say "mypackage.m"), you can load it with Get:
<< "mypackage.m"
Now, there will be a function UnsortedUnion in the Context UU` and it will be accessible globally.
You should also look into the functionality of Needs, but that is a little more advanced in my opinion, so I shall stop here.