I understand that meteor_score from nltk.translate.meteor_score calculates the METEOR-score for one hypothesis sentence based on a list of candidates.
But is there an implementation for calculating the score over an entire corpus as well or a way to do it, similar to the corpus_bleu implementation?
I couldn't find something for this case.
I have created something like this for my project:
#>>> nltk.download()
# Download window opens, fetch wordnet
#>>> from nltk.corpus import wordnet as wn
from nltk.translate.meteor_score import meteor_score
import numpy as np
def corpus_meteor(expected, predicted):
meteor_score_sentences_list = list()
[meteor_score_sentences_list.append(meteor_score(expect, predict)) for expect, predict in zip(expected, predicted)]
meteor_score_res = np.mean(meteor_score_sentences_list)
return meteor_score_res
Related
I am trying to graph variable "u" versus variable "T" for 1<T<1000 (integers). However, the function I have includes both of the variables within an integral so I cannot create an isolated u=f(T) function. My thought process is to manipulate the function so that it is 0=f(T,u) and output a "u" value that minimizes f(T,u) for each T. However, I seem to be struggling a lot with how these variables and functions should be defined. All constants are defined and "E" should be defined through the integration step. The overall function I start with is:
5x10^28=integrate((pi/2)(8m/h^2)(E^0.5)(exp((E-u)/k*T)+1)^-1) from 0 to infinity and with respect to "E"
I am very new to python but the following code is how far I've been able to develop it based on previous forums and video tutorials. Any help is much appreciated!
from scipy.integrate import quad
import numpy as np
import matplotlib.pyplot as plt
import scipy.optimize as spo
m=9.11e-31
h=6.63e-34
k=1.38e-23
T=list(range(1,1001))
def f(E,u):
return (np.pi/2)*(8*m/(h**2))*(E**0.5)*(1/((np.exp((E-u)/k*T)+1)))
Func_Equal_Zero=quad(f,0,np.inf,args=(u,))[0]-5e-28
Start_Guess_T_u=[500,1e-5]
result=spo.minimize(Func_Equal_Zero,Start_Guess_T_u)
plt.plot(T,u)
plt.figure(figsize=(6,6))
plt.xlabel('Temperature (k)')
plt.ylabel('Chemical Potential (J)')
I expected that I could just define the functions including "u" but python does not seem to like what I have tried. I am not sure if any of my other syntax is not correct because I cannot get past its issue with defining "u".
I have a folder where I will upload one file every month. The file will have the same format in every month.
First problem
The idea is to concatenate all the files in this folder into one file. Currently I am hardcoding the filenames (filename[0], filename[1], filename[2]..) but imagine later I will have 50 files, should I explicitly add them to the transform_df decorator? Is there any other method to handle this?
Second problem:
Currently I have let's say 4 files (2021_07, 2021_08, 2021_09, 2021_10) and I want whenever I add the file presenting 2021_12 data to avoid changing the code.
If I add input_5 = Input(path_to_2021_12_do_not_exists) the code will not be run and give an error.
How can I implement the code for future files and let the code ignore the input if it does not exist without manually each month add a new value to my code?
Thank you
# from pyspark.sql import functions as F
from transforms.api import transform_df, Input, Output
from pyspark.sql.functions import to_date, year, col
from pyspark.sql.types import StringType
from myproject.datasets import utils
from pyspark.sql import DataFrame
from functools import reduce
input_dir = '/Company/Project_name/'
prefix_filename = 'DataInput1_'
suffixes = ['2021_07', '2021_08', '2021_09', '2021_10', '2021_11', '2021_12']
filenames = [input_dir + prefix_filename + suffixe for suffixe in suffixes]
#transform_df(
Output("/Company/Project_name/Data/clean/File_concat"),
input_1=Input(filenames[0]),
input_2=Input(filenames[1]),
input_3=Input(filenames[2]),
input_4=Input(filenames[3]),
)
def compute(input_1, input_2, input_3, input_4):
input_dfs = [input_1, input_2, input_3, input_4]
dfs = []
def transformation_input(df):
# some transformation
return df
for input_df in input_dfs:
dfs.append(transformation_input(input_df))
dfs = reduce(DataFrame.unionByName, dfs)
return dfs
This question comes up a lot, the simple answer is that you don't. Defining datasets and executing a build on them are two different steps executed at different stages.
Whenever you commit your code and run the checks, your overall python code is executed during the renderSchrinkwrap stage, except for the compute part. This allows Foundry to discover what datasets exist and publish.
Publishing involves creating your dataset and putting whatever is inside your compute function is published into the jobspec of the dataset, so foundry knows what code to execute whenever you run a build.
Once you hit build on the dataset, Foundry will only pick up whatever is on the jobspec and execute it. Any other code has already run during your checks, and it has run just once.
So any dynamic input/output would require you to re-run checks on your repo, which means that some code change would have had to happen since the Checks is part of the CI process, not part of the build.
Taking a step back, assuming each of your input files has the same schema, Foundry would expect you to have all of those files in the same dataset as append transactions.
This might not be possible though, if for instance, the only indication of the "year" of the data is embedded in the filename, but your sample code would indicate that you expect all these datasets to have the same schema and easily union together.
You can do this manually through the Dataset Preview - just use the Upload File button or drag-and-drop the new file into the Preview window - or, if it's an "end user" workflow, with a File Upload Widget in a Workshop app. You may need to coordinate with your Foundry support team if this widget isn't available.
Bit late to the post although for anyone who is interested in an answer to most of the question. Dynamically determining file names from within a folder is not doable although having some level of dynamic input is possible as follows:
# from pyspark.sql import functions as F
from transforms.api import transform, Input, Output
from pyspark.sql.functions import to_date, year, col
from pyspark.sql.types import StringType
from myproject.datasets import utils
from pyspark.sql import DataFrame
# from functools import reduce
from transforms.verbs.dataframes import union_many # use this instead of reduce
input_dir = '/Company/Project_name/'
prefix_filename = 'DataInput1_'
suffixes = ['2021_07', '2021_08', '2021_09', '2021_10', '2021_11', '2021_12']
filenames = [input_dir + prefix_filename + suffixe for suffixe in suffixes]
inputs = {('input{}'.format(index)): Input(filename) for (index, filename) in enumerate(filenames))}
#transform(
output=Output("/Company/Project_name/Data/clean/File_concat"),
**inputs
)
def compute(output, **kwargs):
# Extract dataframes from input datasets
input_dfs = [dataset_df.dataframe() for dataset_name, dataset_df in kwargs.items()]
dfs = []
def transformation_input(df):
# some transformation
return df
for input_df in input_dfs:
dfs.append(transformation_input(input_df))
# dfs = reduce(DataFrame.unionByName, dfs)
unioned_dfs = union_many(*dfs)
return unioned_dfs
Couple points:
Created dynamic input dict.
That dict is read into the transform using **kwargs.
Using transform decorator not transform_df, we can extract the dataframes.
(not in question) Combine multiple dataframes using union_many function from transforms_verbs library.
I have been trying to compute the wronskian using SymPy, and can not figure out how to use the function. I did look at the program itself but I am very new to python. For functions any sinusoidal is okay. I just want to observe how to use SymPy in this way for future reference. Any help would be great!
~I listed my imports below
import sympy as sp
from scipy import linalg
import numpy as np
sp.init_printing()
I don't this that 'var' is the only thing wrong with what I am inputting.
You have to define the var first. You have not defined it. Also the functions should go in a list.
x = sp.Symbol('x')
## Define your var here
Wronskian_Sol = sp.matrices.dense.wronskian([sp.sin(x), 1-sp.cos(x)**2], var, method="bareiss")
Here is an example in this book "Applied Differntial Equation with Boundary Value Problems" by Vladimir A. Dobrushkin at page 199.
I computed a Wronskian for these three functions using Sympy
x
x*sin(x)
x*cons(x)
import sympy as sp
x = sp.Symbol('x')
var = x
Wronskian_Sol = sp.matrices.dense.wronskian([x, x*sp.cos(x), x*sp.sin(x)], var, method="bareiss")
print(Wronskian_Sol)
print(Wronskian_Sol.simplify())
This gives the output. The first is not simplified, the last one is simplified. You can reduce the first one to simplified version easily by taking the common factor x**3 out which leaves (sin(x)**2 + cos(x)**2) ..and this is nothing but 1.
x**3*sin(x)**2 + x**3*cos(x)**2
x**3
You can confirm the solution by manually taking the determinant of the derivative matrix.
I am having trouble to access the coefficients of a support vector regression model (SVR) in scikit learn when the model is embedded in a pipeline and a grid search.
Consider the following example:
from sklearn.datasets import load_iris
import numpy as np
from sklearn.grid_search import GridSearchCV
from sklearn.svm import SVR
from sklearn.feature_selection import SelectKBest
from sklearn.pipeline import Pipeline
iris = load_iris()
X_train = iris.data
y_train = iris.target
clf = SVR(kernel='linear')
select = SelectKBest(k=2)
steps = [('feature_selection', select), ('svr', clf)]
pipeline = Pipeline(steps)
grid = GridSearchCV(pipeline, param_grid={"svr__C":[10,10,100],"svr__gamma": np.logspace(-2, 2)})
grid.fit(X_train, y_train)
This seems to work fine but when I try to access the coefficient of the best fitting model
grid.best_estimator_.coef_
I get an error message: AttributeError: 'Pipeline' object has no attribute 'coef_'.
I also tried to access the individual steps of the pipeline:
pipeline.named_steps['svr']
but could not find the coefficients there.
Just happened to come across the same problem and this post
had the answer:
grid.best_estimator_ contains an instance of the pipeline, which consists of steps. The last step should always be the estimator, so you should always find the coefficients at:
grid.best_estimator_.steps[-1][1].coef_
I'm using the Wordnet Lemmatizer via NLTK on the Brown Corpus (to determine if the nouns in it are used more in their singular form or their plural form).
i.e. from nltk.stem.wordnet import WordNetLemmatizer
l = WordnetLemmatizer()
I've noticed that even the simplest queries such as the one below takes quite a long time (at least a second or two).
l("cats")
Presumably this is because a web connection must be made to Wordnet for each query?..
I'm wondering if there is a way to still use the Wordnet Lemmatizer but have it perform much faster? For instance, would it help at all for me to download Wordnet on to my machine?
Or any other suggestions?
I'm trying to figure out if the Wordnet Lemmatizer can be made faster rather than trying a different lemmatizer, because I've found it works the best among others like Porter and Lancaster.
It doesn't query the internet, NLTK reads WordNet from your local machine. When you run the first query, NLTK loads WordNet from disk into memory:
>>> from time import time
>>> t=time(); lemmatize('dogs'); print time()-t, 'seconds'
u'dog'
3.38199806213 seconds
>>> t=time(); lemmatize('cats'); print time()-t, 'seconds'
u'cat'
0.000236034393311 seconds
It is rather slow if you have to lemmatize many thousands of phrases. However if you are doing a lot of redundant queries, you can get some speedup by caching the results of the function:
from nltk.stem import WordNetLemmatizer
from functools32 import lru_cache
wnl = WordNetLemmatizer()
lemmatize = lru_cache(maxsize=50000)(wnl.lemmatize)
lemmatize('dogs')
I've used the lemmatizer like this
from nltk.stem.wordnet import WordNetLemmatizer # to download corpora: python -m nltk.downloader all
lmtzr = WordNetLemmatizer() # create a lemmatizer object
lemma = lmtzr.lemmatize('cats')
It is not slow at all on my machine. There is no need to connect to the web to do this.