Ignore errors while loading to Mysql from Pyspark - mysql

In redshift we can have maxerror Option
df.write.option("url", jdbc_url) \
.option("dbtable", tbl) \
.option("tempdir", tmp_folder_rs) \
.option("aws_iam_role", rs_iam) \
.option("extracopyoptions", "**maxerror as 100000** blanksasnull") \
.mode("append") \
.save()
which gives it a tolerance of some records while loading.
Do we have similar options for MYSQL. My write to mysql is as below
job_status_event_req_cols.write.format("jdbc").option("url", "jdbc:mysql://test.us-west-1.vpce.amazonaws.com/test").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "employee").option("user", "writer").option("password", "test").mode('append').save()

Related

Create Azure EventHub via CLI with Capture

Scenario: I am putting together a repeatable script that creates, among other things, an Azure EventHub. My code looks like:
az eventhubs eventhub create \
--name [name] \
--namespace-name [namespace] \
--resource-group [group] \
--status Active \
--enable-capture true \
--archive-name-format "{Namespace}/{EventHub}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}/{PartitionId}" \
--storage-account [account] \
--blob-container [blob] \
--capture-interval 300 \
--partition-count 10 \
--skip-empty-archives true
If I run the code as written, I get a "Required property 'name' not found in JSON. Path 'properties.captureDescription.destination', line 1, position 527."
However, if I remove the --enable-capture true parameter, the EventHub is created, albeit with Capture not enabled. If I enable Capture, none of the capture-related parameters other than the interval are set.
Is there a typo in there that I'm not seeing?
Try providing the --destination-name.
az eventhubs eventhub create --name
--namespace-name
--resource-group
[--archive-name-format]
[--blob-container]
[--capture-interval]
[--capture-size-limit]
[--destination-name]
[--enable-capture {false, true}]
[--message-retention]
[--partition-count]
[--skip-empty-archives {false, true}]
[--status {Active, Disabled, SendDisabled}]
[--storage-account]

median in django annotation

Im using MySQL server v5.8 as database, how to find median price in query like this:
price_query = Product.objects \
.filter(price_query) \
.annotate(dt=Trunc('StartDate', frequency)) \
.values('dt') \
.annotate(avg_price=Avg('Price'), std_price=StdDev('Price'),
count=Count('Price'), max_price=Max('Price'),
min_price=Min('Price'), median='???') \
.order_by('dt')
response is look like this
{"date":"2021-05-01T00:00:00Z","avg_price":4326.666666666667,"std_price":20.548046676563168,"min_price":4300.0, "max_price":4350.0,"count":3}, {...}, {...}
Any help is highly appreciated.

how to link static jemalloc for mysql5.5

I want to link static jemalloc library for oracle mysql5.5 .If mysql5.1 ,i can do it :
LDFLAGS=” -lrt -ljemalloc -lstdc++ ”
./configure –prefix=/usr/soft/install/mysql \
–with-mysqld-user=mysql \
–with-plugins=partition,blackhole,csv,heap,innobase,myisam,myisammrg \
–with-charset=utf8 \
–with-collation=utf8_general_ci \
–with-extra-charsets=gbk,gb2312,utf8,ascii \
–with-big-tables \
–with-fast-mutexes \
–with-zlib-dir=bundled \
–enable-assembler \
–enable-profiling \
–enable-local-infile \
–enable-thread-safe-client \
–with-readline \
–with-pthread \
–with-embedded-server \
–with-mysqld-ldflags=-all-static \
–without-query-cache \
–without-geometry \
–without-debug \
–without-ndb-debug
make && make install
But i can not find a cmake's option with it in mysql5.5.
What can i do?

MooTools build hash in 1.2.4.4

We are trying to upgrade our MooTools installation from 1.2.4 to 1.2.6. The original developer included a "more" file with optional plugins, but because it is compressed we can't tell what was included in that file. I'd rather not hunt and pick through the code.
I noticed the compressed more file has a build hash in the header (6f6057dc645fdb7547689183b2311063bd653ddf). The 1.4 builder located here will let you just append that hash to the url and create a build. It doesn't seem the 1.2 version supports that functionality.
Is there an easy way to determine from the hash or the compressed file what plugins are included in this 1.2 build?
AFAIK there's no way to get the list of plugins directly from the build hash. But if you have access to a UNIX shell, save the following shell script as find_plugins.sh:
#!/bin/sh
for PLUGIN in \
More Lang Log Class.Refactor Class.Binds Class.Occlude Chain.Wait \
Array.Extras Date Date.Extras Hash.Extras String.Extras \
String.QueryString URI URI.Relative Element.Forms Elements.From \
Element.Delegation Element.Measure Element.Pin Element.Position \
Element.Shortcuts Form.Request Form.Request.Append Form.Validator \
Form.Validator.Inline Form.Validator.Extras OverText Fx.Elements \
Fx.Accordion Fx.Move Fx.Reveal Fx.Scroll Fx.Slide Fx.SmoothScroll \
Fx.Sort Drag Drag.Move Slider Sortables Request.JSONP Request.Queue \
Request.Periodical Assets Color Group Hash.Cookie IframeShim HtmlTable \
HtmlTable.Zebra HtmlTable.Sort HtmlTable.Select Keyboard Keyboard.Extras \
Mask Scroller Tips Spinner Date.English.US Form.Validator.English \
Date.Catalan Date.Czech Date.Danish Date.Dutch Date.English.GB \
Date.Estonian Date.German Date.German.CH Date.French Date.Italian \
Date.Norwegian Date.Polish Date.Portuguese.BR Date.Russian Date.Spanish \
Date.Swedish Date.Ukrainian Form.Validator.Arabic Form.Validator.Catalan \
Form.Validator.Czech Form.Validator.Chinese Form.Validator.Dutch \
Form.Validator.Estonian Form.Validator.German Form.Validator.German.CH \
Form.Validator.French Form.Validator.Italian Form.Validator.Norwegian \
Form.Validator.Polish Form.Validator.Portuguese \
Form.Validator.Portuguese.BR Form.Validator.Russian \
Form.Validator.Spanish Form.Validator.Swedish Form.Validator.Ukrainian
do
grep -q -F $PLUGIN $1 && echo $PLUGIN
done
Then run it like this passing the filename of your MooTools More file as first argument:
sh find_plugins.sh mootools-more.js
It will print out a list of all plugin names found in the JS code. That should get you started.

what are flags in mysql?

I just wanna ask, what are those things called as "flags" in mysql ?
What are they, kindly explain and how they are used.
mysql flags are used by the compiler and set the enviroment you wish to work in.
There's flags such as:
'--with-archive-storage-engine'
wgich does what it states, it would start mysql process but also start the system in archive able content mode
Some more examples:
'--prefix=/usr/local/mysql' \
'--disable-dependency-tracking' \
'--enable-assembler' \
'--localstatedir=/usr/local/mysql/data' \
'--libexecdir=/usr/local/mysql/bin' \
'--libdir=/usr/local/mysql/lib' \
'--enable-local-infile' \
'--enable-shared' \
'--enable-thread-safe-client' \
'--with-archive-storage-engine' \
'--with-big-tables' \
'--with-comment=MySQL Community Server (GPL)' \
'--with-extra-charsets=complex' \
'--with-innodb' \
'--with-mysqld-ldflags=-all-static' \
'--with-readline' \
'--with-server-suffix=-standard' \
'--with-unix-socket-path=/tmp/mysql.sock' \
'--with-zlib-dir=bundled' \
'--without-debug' \
'--without-docs' \
'--without-man'