Related
Good morning all!!
Yesterday I was looking for a function that was able to write a single file CSV with pyspark in Azure Databricks but I did not find anything. So I've built my own function and I wanted to share my solution with the community and if it's possible create like a thread with different solutions for the same problem.
Sorry, because I commented the code in Spanish but basically the function does:
Save the dataframe you've created into a new directory (which is allocated inside the path you've defined and begin with 'temp_') and save the partition there using "coalesce(1)"
Rename the CSV file as you want and moves it to the desired path
Delete de temporary file
That's all! You have your unique CSV file
def escribe_fichero_unico(dataframe, path, file_name, file_format = 'csv'):
"""
Definición: (1) Genera carpeta temporal para guardar particiones que Spark genera por defecto
a la hora de guardar archivos, (2) Une todas las particiones en un único archivo, (3) Mueve este
archivo al directorio anterior y (4) Borra la carpeta temporal
Parámetros:
dataframe: dataframe que quieras guardar como fichero único
file_name: en formato string escribe nombre del archivo
file_format: en formato string escribe 'csv' o 'parquet'
path: en formato string escribe el path donde quieres guardar el csv
"""
import os
# 1) Guardamos el dataframe creando una carpeta temporal que guarda todas las particiones
path_temp = path + 'temp_' + file_name + '_trash'
if file_format == 'csv':
dataframe.coalesce(1).write.format('csv').mode('overwrite') \
.options(header="true", schema="true", delimiter=";") \
.save(path_temp)
else if file_format == 'parquet':
dataframe.coalesce(1).write.format("parquet").mode("overwrite") \
.save(path_temp)
# 2)Une todas las particiones en un único archivo
file_part = [file.path for file in dbutils.fs.ls(path_temp) if os.path.basename(file.path).startswith('part')][0]
# 3) Mueve este archivo al directorio anterior
dbutils.fs.mv(file_part, path + file_name + '.' + file_format)
# 4) Borra la carpeta temporal
dbutils.fs.rm(path_temp, True)
I hope this work for you as well :)
I'm starting to work with PowerShell and my first task is to fix a problem with a script that already runs in the production environment.
This script is called via Webhook and it receives a parameter that the webhook passes to it.
I need to run this script inside PowerShell ISE to be able to debug it but I don't know how to fill the variables that are normally filled when it is called by Webhook.
Here is the beginning of the code where the variables are filled in, can someone give me a tip on how to fill the variable "WebHookData"..?
Thanks in advance.
I've tried to do this but it didn't work...
Sorry for putting images instead of the code, but for some reason I can't post the code.
This is the JSON that I use..
{"source":"la-draft-clipboard","value":[{"tokenKey":"8EAD3F03-E08F-4D58-8B1A-2AB8BD2F25DB","type":"literal","tokenExpression":"{"},{"tokenKey":"A7596123-17DF-49A9-AC18-1196A4CD457E","type":"new_line","tokenExpression":"\n"},{"tokenKey":"36DF511D-C1A9-4BC8-B2E9-37BCA058FB78","type":"literal","tokenExpression":" \"AutomationAccountName\": \"proj-00016-automation-account\","},{"tokenKey":"918137AE-EC61-4B77-A5F2-B527E2D4E3C9","type":"new_line","tokenExpression":"\n"},{"tokenKey":"DCC2D1C1-14F0-4869-A44C-08F8AB35B0B3","type":"literal","tokenExpression":" \"BeginPeakTime\": \"7:00\","},{"tokenKey":"61F7441B-0688-4AD2-A1A5-086C4F7F6D1E","type":"new_line","tokenExpression":"\n"},{"tokenKey":"2F3DD3CA-BD83-46EF-9529-C890C2E31CAF","type":"literal","tokenExpression":" \"ConnectionAssetName\": \"AzureRunAsConnection\","},{"tokenKey":"C6DD6FD0-E99A-48A8-96AA-3974D66FD9BD","type":"new_line","tokenExpression":"\n"},{"tokenKey":"A4E7A469-D08A-4C5A-8C6B-06E58996A0EC","type":"literal","tokenExpression":" \"EndPeakTime\": \"17:00\","},{"tokenKey":"E67547BC-98BB-4749-A84E-A36B761EE504","type":"new_line","tokenExpression":"\n"},{"tokenKey":"727D64BD-906C-4DA3-84C5-44F3054B2DEB","type":"literal","tokenExpression":" \"HostPoolName\": \"VDI-POOL-001\","},{"tokenKey":"92AFEBB8-4307-42C2-8BD0-C55ACC848940","type":"new_line","tokenExpression":"\n"},{"tokenKey":"F37993F9-1471-4E58-B43F-9BB08C4D4A03","type":"literal","tokenExpression":" \"LimitSecondsToForceLogOffUser\": 0,"},{"tokenKey":"8B2517D1-046E-43EF-BF75-B1EC5F31B83D","type":"new_line","tokenExpression":"\n"},{"tokenKey":"7464316E-6A8D-4F82-B269-95FF76A69014","type":"literal","tokenExpression":" \"LogOffMessageBody\": \"Salve seus trabalhos! Em aproximadamente 15 minutos, este terminal virtual será desligado automaticamente devido às políticas de otimização de custos da companhia. Caso seja necessário continuar suas atividades, um novo terminal poderá ser acessado após este período.\","},{"tokenKey":"7328955E-0025-4AA1-A0AE-CDAFA4238927","type":"new_line","tokenExpression":"\n"},{"tokenKey":"384AF3CF-CA86-4820-A5E1-230C09909662","type":"literal","tokenExpression":" \"LogOffMessageTitle\": \"ATENÇÃO!!!\","},{"tokenKey":"5E2EBD78-8599-487F-8DC5-CF9699595DDD","type":"new_line","tokenExpression":"\n"},{"tokenKey":"B7E409AF-A5AE-4622-A45E-5982FD15B03E","type":"literal","tokenExpression":" \"MaintenanceTagName\": \"NO_TAG\","},{"tokenKey":"3F9BF963-790D-45B1-9F04-D71A2B7C84DC","type":"new_line","tokenExpression":"\n"},{"tokenKey":"B6E94E37-69C0-4BF8-AE69-CD7B4EA9CB83","type":"literal","tokenExpression":" \"MinimumNumberOfRDSH\": 20,"},{"tokenKey":"00A1D37B-F82B-42F6-B792-75B39EBD6A83","type":"new_line","tokenExpression":"\n"},{"tokenKey":"F41B0C75-4541-4772-BF30-2D4F6DF045C6","type":"literal","tokenExpression":" \"ResourceGroupName\": \"proj-00016-wvd-rg\","},{"tokenKey":"FE6FC329-DC12-4782-83CE-F48BDC6B74B5","type":"new_line","tokenExpression":"\n"},{"tokenKey":"785500F8-3D71-4D91-AADA-D6ABF1EFD66B","type":"literal","tokenExpression":" \"ResourceGroupNameAutomation\": \"proj-00016-automation-rg\","},{"tokenKey":"BD3331BF-3BF9-4B9E-B9B8-C03E448B2D85","type":"new_line","tokenExpression":"\n"},{"tokenKey":"25586050-62A0-4CAF-81FD-C5770DF20B63","type":"literal","tokenExpression":" \"RunbookLogoffShutdown\": \"ARMLogoffAndShutdown\","},{"tokenKey":"C4B9E432-C41D-4374-9531-F2AEFDD51267","type":"new_line","tokenExpression":"\n"},{"tokenKey":"0155B6AB-7CAB-4C4E-BB1F-A643D9B0575B","type":"literal","tokenExpression":" \"SessionThresholdPerCPU\": 0.75,"},{"tokenKey":"3EAA1C7E-0119-40B9-9AF8-85D10E0FA3FD","type":"new_line","tokenExpression":"\n"},{"tokenKey":"2D904698-1386-47D7-9513-7CEE702BA0D3","type":"literal","tokenExpression":" \"TimeDifference\": \"-3:00\""},{"tokenKey":"40D497B6-AAED-4334-81C7-10B8C6745DE0","type":"new_line","tokenExpression":"\n"},{"tokenKey":"18EB90AF-25D4-4956-8A85-41BA555C6A95","type":"literal","tokenExpression":"}"}]}
Based on the JSON you've posted and the parts of the code we can see in the screenshot, give the following mock object a try:
$mockWebhookPayload = [pscustomobject]#{
WebhookName = 'NameOfWebhookGoesHere'
RequestHeader = #{ 'Content-Type' = 'application/json' }
RequestBody = #'
{
"AutomationAccountName": "proj-00016-automation-account",
"BeginPeakTime": "7:00",
"ConnectionAssetName": "AzureRunAsConnection",
"EndPeakTime": "17:00",
"HostPoolName": "VDI-POOL-001",
"LimitSecondsToForceLogOffUser": 0,
"LogOffMessageBody": "Salve seus trabalhos! Em aproximadamente 15 minutos, este terminal virtual será desligado automaticamente devido às políticas de otimização de custos da companhia. Caso seja necessário continuar suas atividades, um novo terminal poderá ser acessado após este período.",
"LogOffMessageTitle": "ATENÇÃO!!!",
"MaintenanceTagName": "NO_TAG",
"MinimumNumberOfRDSH": 20,
"ResourceGroupName": "proj-00016-wvd-rg",
"ResourceGroupNameAutomation": "proj-00016-automation-rg",
"RunbookLogoffShutdown": "ARMLogoffAndShutdown",
"SessionThresholdPerCPU": 0.75,
"TimeDifference": "-3:00"
}
'#
}
& .\path\to\webhook-script.ps1 -WebHookData $mockWebhookPayload
I'm trying to import a .json into a database using MySQL workbench, but the "Table data import wizard" always end up with 0 records.
here's the first line of my .json:
{
"Title":"Fort de Villiers",
"Keywords":"Histoire",
"Descripion":"Le fort de Villiers est un ancien fort du système défensif de Paris, situé sur la commune de Noisy-le-Grand, dans le département français de Seine-Saint-Denis. Construit à la fin du XIXᵉ siècle, il faisait partie du système Séré de Rivières, destiné a établir un rideau défensif autour de Paris.",
"Images":["https://lh3.googleusercontent.com/proxy/J-7ut85_OfCJn33_EZcooBkGyjxGBXt5CifUJ_bGzHaZUHlT4XVYwNaIKpZ4hScn3ICcUMI12RZ4xzTxUGmN1LCK694hynag4zh68dLypDwQ_Gy9NwCIFweGTjkh3Wft4IMr0No1tFWbTQ6oBnmLgRaomsWzvoJrTa8=w296-h202-n-k-no","https://lh5.googleusercontent.com/p/AF1QipNs2wi72SiKVlLr1WHIPfAaATrXOfhswYWEIaOw=w296-h202-n-k-no","https://lh5.googleusercontent.com/p/AF1QipMIY16VgeazmwQAP7PB5biaq5p-9uE3OZ-zVYZB=w296-h202-n-k-no"],
"Lattitude":"48.833573",
"Longitude":"2.5463771",
"Map_url":"https://www.google.com/maps/place/Fort+de+Villiers/#48.833573,2.5463771,15z/data=!4m5!3m4!1s0x47e60e0d72beac57:0xd4b8549d2165eef7!8m2!3d48.8335733!4d2.5551319"
}
when i try to import it, i got those fields proposed:
Title : Text
Description : Text
Longitude : Double
Images : json
Map_url : Text
Keywords : Text
Lattitude : Double
With some research i found out that it may be that the "text" columns countains characters that doesn't fill the requiring format, but i didn't manage to get the right format.
Do i need to modify my .json to get a correct utf8 format?
I'm actually using mysql 8.
I need to write this description in the manifest.json:
"description" : "Somos la única empresa agroindustrial del país presente en toda la cadena de valor granaria."
The problem is when I deploy the app, in google chrome console the accents marks are broken
"description" : "Somos la única empresa agroindustrial del paÃs presente en toda la cadena de valor granaria."
How can I fix that?
You will need to modify your web server config, if you are running Apache you can create or update your .htaccess and add the following:
AddDefaultCharset utf-8
AddType 'application/json; charset=UTF-8' .json
You should change the Charset to UTF-8, but in this case it's not a problem of the file encoding.
I use TRESTClient, TRESTRequest, FRESTResponse and RESTResponseDataSetAdapter like in RESTDebugger (Delphi Tokyo) and it works fine but for one JSONValue the webservice send me some HTML mix and It do not display the right thing in my TFDMemTable.
I make the same test with JSONViewer and it make the same things
Is there a way to modify my JSONStream just before it start to parse my RestReponse ?
Why it don't display all my JSONString for dev_msg_attention_to_garage ?
{"id_commande":"15672","id_devis":"16347","status":"CONFIRM","res_codres":"IDG","nom_client":"TESTRDV","prenom_client":"toto","mail":"test#test.fr","tel_port":"0606060606","date_prise_rdv":"2017-09-25 09:25:43.281269","date_rdv":"2017-09-27","heure_rdv":"17:30-18:00","immatriculation":"EC620XP ","type_prestation":"FREINAGE","famille":"Disques et plaquettes avant","type_mine":"","codmar":"PEUG","numord":"9938","marque":"PEUGEOT","marque_modele":"PEUGEOT 206 1.1 60CV","cnit":"MPE1002K6315","aaa_caliloca":null,"aaa_cape":null,"aaa_ccreditbail":null,"aaa_canton":null,"aaa_carrosserie":"BERLINE","aaa_carrosseriecg":"CI","aaa_catclient":null,"aaa_catclientvo":null,"aaa_catvendeur":null,"aaa_co2":"154","aaa_codifvin":"VF32AHFZE40061368","aaa_consexurb":"5.1","aaa_consmixte":"6.2","aaa_consurb":"8.2","aaa_couleurvehic":"MARRON","aaa_cstrvincg":null,"aaa_cylindree":"1124","aaa_date1ercir":"1998-10-15","aaa_datedcg":"2016-06-09","aaa_datedcgachat":null,"aaa_datedcgpr":null,"aaa_datedcgprachat":null,"aaa_depart":null,"aaa_departement":null,"aaa_depollution":"NON","aaa_empat":"245.0","aaa_empreintesol":null,"aaa_energie":"ESSENCE","aaa_gammemarche":null,"aaa_genrev":"VP","aaa_genrevcg":"VP","aaa_hauteur":"144","aaa_immatsiv":"EC620XP","aaa_largeur":"165","aaa_longueur":"384","aaa_marque":"PEUGEOT","aaa_marquecarros":"PEUGEOT","aaa_modeinject":"AUTR. CARBURATIONS","aaa_moderefroid":null,"aaa_modele":"206","aaa_modeleetude":"206","aaa_modeleprf":"206","aaa_mouvmt":null,"aaa_nserie":"40061368","aaa_nsiren":"000000000","aaa_nbcylind":"4","aaa_nbdecibels":null,"aaa_nbmain":null,"aaa_nbplass":"5","aaa_nbportes":"5","aaa_nbpropriet":null,"aaa_nbsoupape":"2","aaa_nbvitesse":"5","aaa_nbvolume":"2","aaa_nuancecouleur":null,"aaa_poidsvide":"910","aaa_prixvehic":"0","aaa_propulsion":"AVANT","aaa_ptr":"1405","aaa_ptrprf":"0192","aaa_puisch":"60","aaa_puisfisc":"5","aaa_rappuispoids":null,"aaa_regionadmin":null,"aaa_regmdecibels":null,"aaa_statutimmat":null,"aaa_temoincession":null,"aaa_tpboitevit":"MECANIQUE","aaa_transmiss":null,"aaa_turbocompr":"NON TURBO","aaa_type":"MPE1002K6315","aaa_typevincg":"2AHFZE","aaa_version":"XR\/PRESE","aaa_voiear":null,"aaa_voieav":null,"aaa_datews":"2017-06-16","aaa_3lnom":"fff","veh_codmar":null,"veh_numord":null,"veh_ktypenr":null,"aaa_pneus":"175\/65 R14 82 T, 165\/70 R13 79 T","aaa_typevarversprf":"2AHFZE","aaa_codemoteur":"HFZ","aaa_puiskw":"44","total_ht":"141.58","total_tva":"28.32","total_ttc":"169.90","total_ttc_max":"169.90","total_ttc_min":"169.90","dev_msg_attention_to_garage":" \r\n \r\n \r\n attention\r\n \r\n ATTENTION Pour ce rendez-vous web nous avons identifié\r\n plusieurs références constructeur possible :<\/br\/>\r\n 4249A5, <b>424995<\/b><br\/> \r\n <br \/>Vous devez vous assurer que la\r\n référence indiquée correspond bien à\r\n celle montée sur le véhicule du client <br \/>Le\r\n prix final à payer par le client doit être de 169,90 \u20AC TTC \r\n \r\n \r\n\t\r\n \/ Le devis est en pi\u00E8ce budget","dev_msg_from_customer":"test message;\r\n","id":"15672","created_at":"2017-09-25 09:25:43.460187","updated_at":"2017-09-25 09:26:10.621387","coupon":[],"promotion_globale":[],"lignes_pieces":[{"codfam":"FDI","ssfam":"AV","designation_piece":"DISQUE DE FREIN AV PLEIN. DIAMETRE= 247X10MM","ref_constructeur":"424995","quantity":"2.00","remise":"0.00","pu":"0.00","taux_tva":null},{"codfam":"FPL","ssfam":"AV","designation_piece":"PLAQUETTES DE FREIN AV. MONTAGE BOSCH","ref_constructeur":"425320","quantity":"1.00","remise":"0.00","pu":"0.00","taux_tva":null}],"lignes_mo":[{"type_ligne":"FOI","designation_mo":"Forfait Isotech ","ref_operation_adauto":"FISOAD16990","quantity":"1.00","taux_horaire":"141.58","remise":null,"code_taux_horaire":null,"taux_tva":1.2,"montant_ht_ligne_apres_remise":141.58}],"deduction_rdv_en_ligne_ttc":"16.99","montant_a_regler_par_client":152.91,"pourcentage_remise_calculee_pieces":0,"total_remises":0,"total_ht_avant_remises":141.58,"pourcentage_remises":0,"pourcentage_remise_calculee_mo":0}
You are being presented with a limitation of Windows' Tree View. From the documentation:
Note that although the tree-view control allows any length string to be stored as item text, only the first 260 characters are displayed.
(Emphasis mine)
The quick answer is that since this control wraps Delphi's TTreeView implementation, it also inherits this limitation, and thus there's not too much you can do to overcome it, besides either implementing custom drawing or finding another control without this limitation.