We are using gradle.plugin.org.detoeuf:swagger-codegen-plugin.
We want to change the content of the build.gradle file in the output directory.
We added the gradle.build file to the .swagger-codegen-ignore but BOTH .swagger-codegen-ignore file and gradle.build file are re-created every time we call the swagger task.
Our swagger section looks like this
swagger {
inputSpec = "${project.projectDir}/swagger/backoffice-service-api-swagger.json"
outputDir = file("${project.projectDir}/../backoffice-service-api-client/")
lang = 'java'
additionalProperties = [
'invokerPackage' : 'com.aaa.bbb.backoffice.service',
'modelPackage' : 'com.aaa.bbb.backoffice.service.model',
'apiPackage' : 'com.aaa.bbb.backoffice.service.api',
'dateLibrary' : 'joda',
'groupId' : 'com.aaa.bbb',
'artifactId' : 'backoffice-service-api-client',
'artifactVersion' : '1.0.0',
'hideGenerationTimestamp': 'true',
'dateLibrary' : 'java8'
]
systemProperties = ["apiTests" : "false"]
}
.swagger-codegen-ignore file looks like this -
# Swagger Codegen Ignore
# Generated by swagger-codegen https://github.com/swagger-api/swagger-codegen
# Use this file to prevent files from being overwritten by the generator.
# The patterns follow closely to .gitignore or .dockerignore.
# As an example, the C# client generator defines ApiClient.cs.
# You can make changes and tell Swagger Codgen to ignore just this file by uncommenting the following line:
#ApiClient.cs
# You can match any string of characters against a directory, file or extension with a single asterisk (*):
#foo/*/qux
# The above matches foo/bar/qux and foo/baz/qux, but not foo/bar/baz/qux
# You can recursively match patterns against a directory, file or extension with a double asterisk (**):
#foo/**/qux
# This matches foo/bar/qux, foo/baz/qux, and foo/bar/baz/qux
# You can also negate patterns with an exclamation (!).
# For example, you can ignore all files in a docs folder with the file extension .md:
#docs/*.md
# Then explicitly reverse the ignore rule for a single file:
#!docs/README.md
build.gradle
You can add ignoreFileOverride option in additionalProperties as below. The files provided in ignoreFileOverride option do not exists in project, then swagger-codegen will generate them and if files exist in project then swagger-codegen will ignore them.
swagger {
inputSpec = "${project.projectDir}/swagger/backoffice-service-api-swagger.json"
outputDir = file("${project.projectDir}/../backoffice-service-api-client/")
lang = 'java'
additionalProperties = [
'invokerPackage' : 'com.aaa.bbb.backoffice.service',
'modelPackage' : 'com.aaa.bbb.backoffice.service.model',
'apiPackage' : 'com.aaa.bbb.backoffice.service.api',
'dateLibrary' : 'joda',
'groupId' : 'com.aaa.bbb',
'artifactId' : 'backoffice-service-api-client',
'artifactVersion' : '1.0.0',
'hideGenerationTimestamp': 'true',
'dateLibrary' : 'java8',
'ignoreFileOverride' : '.swagger-codegen-ignore,build.gradle'
]
systemProperties = ["apiTests" : "false"]
}
Related
I want to Load Multiple CSV files matching certain names into a dataframe. Currently i am looping through the whole folder and creating a list of filenames and then loading those csv's into the dataframe list and then concatenating that dataframe.
The approach i want to use (if possible) is to bypass all the code and read all files in a one liner kind of approach.
I know this can be done easily for single level of subfolders, but my subfolder structure is as follows
Root Folder
|
Subfolder1
|
Subfolder 2
|
X01.csv
Y01.csv
Z01.csv
|
Subfolder3
|
Subfolder4
|
X01.csv
Y01.csv
|
Subfolder5
|
X01.csv
Y01.csv
I want to read all "X01.csv" files while reading from Root Folder.
Is there a way i can read all the required files in code something like the below
filepath = "rootpath" + "/**/X*.csv"
df = spark.read.format("com.databricks.spark.csv").option("recursiveFilelookup","true").option("header","true").load(filepath)
This code works fine for single level of subfolders, is there any equivalent of this for multi level folders ? i thought the "recursiveFilelookup" option would look across all levels of subfolders, but apparently this is not the way it works.
Currently i am getting a
Path not found ... filepath
exception
any help please
Have you tried using the glob.glob function?
You can use it to search for files that match certain criteria inside a root path, and pass the list of files it finds to spark.read.csv function.
For example, I've recreated the folder structure from your example inside a Google Colab environment:
To get a list of all CSV files matching the criteria you've specified, you can use the following code:
import glob
rootpath = './Root Folder/'
# The following line of code looks through all files
# inside the rootpath recursively, trying to match the
# pattern specified. In this case, it tries to find any
# CSV file that starts with the letters X, Y, or Z,
# and ends with 2 numbers (ranging from 0 to 9).
glob.glob(rootpath + "**/[X|Y|Z][0-9][0-9].csv", recursive=True)
# Returns:
# ['./Root Folder/Subfolder5/Y01.csv',
# './Root Folder/Subfolder5/X01.csv',
# './Root Folder/Subfolder1/Subfolder 2/Y01.csv',
# './Root Folder/Subfolder1/Subfolder 2/Z01.csv',
# './Root Folder/Subfolder1/Subfolder 2/X01.csv']
Now you can combine this with spark.read.csv capability of reading a list of files to get the answer you're looking for:
import glob
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
rootpath = './Root Folder/'
spark.read.csv(glob.glob(rootpath + "**/[X|Y|Z][0-9][0-9].csv", recursive=True), inferSchema=True, header=True)
Note
You can specify more general patterns like:
glob.glob(rootpath + "**/*.csv", recursive=True)
To return a list of all csv files inside any subdirectory of rootpath.
Additionally, to consider only the immediate subdirectories files, you could use something like:
glob.glob(rootpath + "*.csv", recursive=True)
Edit
Based on your comments to this answer, does something like this works on Databricks?
from notebookutils import mssparkutils as ms
# databricks has a module called dbutils.fs.ls
# that works similarly to mssparkutils.fs, based on
# the following page of its documentation:
# https://docs.databricks.com/dev-tools/databricks-utils.html#ls-command-dbutilsfsls
def scan_dir(
initial_path: str,
search_str: str,
account_name: str,
):
"""Scan a directory and subdirectories for a string.
Parameters
----------
initial_path : str
The path to start the search. Accepts either a valid container name,
or the entire connection string.
search_str : str
The string to search.
account_name : str
The name of the account to access the container folders.
This value is only used, when the `initial_path`, doesn't
conform with the format: "abfss://<initial_path>#<account_name>.dfs.core.windows.net/"
Raises
------
FileNotFoundError
If the `initial_path` informed doesn't exist.
ValueError
If `initial_path` is not a string.
"""
if not isinstance(initial_path, str):
raise ValueError(
f'`initial_path` needs to be of type string, not {type(initial_path)}'
)
elif not initial_path.startswith('abfss'):
initial_path = f'abfss://{initial_path}#{account_name}.dfs.core.windows.net/'
try:
fdirs = ms.fs.ls(initial_path)
except Py4JJavaError as exc:
raise FileNotFoundError(
f'The path you informed \"{initial_path}\" doesn\'t exist'
) from exc
found = []
for path in fdirs:
p = path.path
if path.isDir:
found = [*found, *scan_dir(p, search_str)]
if search_str.lower() in path.name.lower():
# print(p.split('.net')[-1])
found = [*found, p.replace(path.name, "")]
return list(set(found))
Example:
# Change .parquet to .csv
spark.read.parquet(*scan_dir("abfss://CONTAINER_NAME#ACCOUNTNAME.dfs.core.windows.net/ROOT/FOLDER/", ".parquet"))
This method above worked for on Azure Synapse:
Scenario:
I have inspec profile-A(10 controls), Profile-B(15 controls), Profile-C(5 controls)
Profile-A depends on Profile-B and Profile-C.
I have a file in Profile-A which I am prasing with inspec.profile.file('test.json') and executing the 10 controls in the same profile.
I have to pass the same file to Profile-B and Profile-C so that I can execute the other set of tests in each profile as part of the profile dependency
I am able to successfully parse the test.json file in profile-A as the file is in correct folder path
myjson = json(content: inspec.profile.file('test.json'))
puts myjson
I have followed the inspec documentation to set up the profile dependency and inputs to the dependant profiles.
https://docs.chef.io/inspec/inputs/
Issue:
Issue is that I am able to pass a single Input values (like string, array etc..) to the dependent profiles but not able to pass the entire json file so that it will parse and the controls will be executed.
I have tried the following in the profile metadata file
# ProfileB inspec.yml
name: profile-b
inputs:
- name: file1
- name: file2
# wrapper inspec.yml
name: profile-A
depends:
- name: profile-b
path: ../profile-b
inputs:
- name: file1
value: 'json(content: inspec.profile.file('test.json'))'
profile: profile-b
- name: file2
value: 'FILE.read('/path/to/test.json')'
profile: profile-b
Error:
when I try to load the file1 and file2 in profile-b with the following
jsonfile1 = input('file1')
jsonfile2 = input('file2')
puts jsonfile1
puts jsonfile2
error - no implicit conversion of nil to integer
Goal:
I should be able to pass the file from profile-A to profile-B or profile-C so that the respective dependent profile controls are execute.
I have a configuration file with:
{path, "/mnt/test/"}.
{name, "Joe"}.
The path and the name could be changed by a user. As I know, there is a way to save those variables in a module by usage of file:consult/1 in
-define(VARIABLE, <parsing of the config file>).
Are there any better ways to read a config file when the module begins to work without making a parsing function in -define? (As I know, according to Erlang developers, it's not the best way to make a complicated functions in -define)
If you need to store config only when you start the application - you may use application config file which is defined in 'rebar.config'
{profiles, [
{local,
[{relx, [
{dev_mode, false},
{include_erts, true},
{include_src, false},
{vm_args, "config/local/vm.args"}]
{sys_config, "config/local/yourapplication.config"}]
}]
}
]}.
more info about this here: rebar3 configuration
next step to create yourapplication.config - store it in your application folder /app/config/local/yourapplication.config
this configuration should have structure like this example
[
{
yourapplicationname, [
{path, "/mnt/test/"},
{name, "Joe"}
]
}
].
so when your application is started
you can get the whole config data with
{ok, "/mnt/test/"} = application:get_env(yourapplicationname, path)
{ok, "Joe"} = application:get_env(yourapplicationname, name)
and now you may -define this variables like:
-define(VARIABLE,
case application:get_env(yourapplicationname, path) of
{ok, Data} -> Data
_ -> undefined
end
).
I have an info.json file with the line:
"install_requires" : {"" : []},
Assuming that this is where I put packages that my program relies on, how would I format it if I want it to rely on
python-redmine
version 0.8.3
Would it possibly be
"install_requires" : {"python-redmine" : [0.8.3]},
Should be
"install_requires" : {"":["python-redmine==0.8.3"]},
By default phpsh uses system's default PHP, my system it's /usr/local/bin/php, which is php 5.2. How can I run phpsh with my custom php path — /srv/bin/php?
It's not supported by default, but you can quickly add.
It by applying this patch to your phpsh.py in the src folder:
--- phpsh.py 2011-05-13 18:16:32.000000000 -0400
+++ phpsh.py 2013-12-05 14:50:11.906673382 -0500
## -253,6 +253,7 ##
def __init__(self):
self.config = ConfigParser.RawConfigParser({
"UndefinedFunctionCheck": "yes",
+ "PathToBinary" : None,
"Xdebug" : None,
"DebugClient" : "emacs",
"ClientTimeout" : 60,
## -388,6 +389,8 ##
except Exception, msg:
self.print_error("Failed to load config file, using default "\
"settings: " + str(msg))
+ if self.config.get_option("General", "PathToBinary"):
+ os.environ['PATH'] = self.config.get_option("General", "PathToBinary") + ':' + os.environ['PATH']
if self.with_xdebug:
xdebug = self.config.get_option("Debugging", "Xdebug")
if xdebug and xdebug != "yes":
If you want to modify an already installed version, find your pythons site-pages folder and apply the patch on both init.py and phpsh.py in that folder.
This will add a new configuration variable so in phpsh/config (in /etc/phpsh/config if installed as root, or ~/.phpsh/config if user). In there you can specify the path to your php binary
PathToBinary: /srv/bin
This is just the path where the binary should be found, not the path binary itself, i.e. /srv/bin/php will not work.