My python program is meant to query my MySQL database for a record. The record exists in the database and contains data but the program returns null values. The table that gets queried is titled Market. In that table there is a column titled market_cap and a column titled volume. When I use MySQLWorkbench to query my database, the result shows that there is data in the columns. However, the program receives null.
Attached are two images (links, because I need to earn 10 reputation points to embed images in a post):
MySql database column image
shows a view of the column in my database that is having issues.
From the image, you can see that the data I need exists in my database.
Code with results from Pycharm debugger
Before running the debugger, I set a breakpoint right after the line where
the code queries the database for an object. Image two shows the output I
received when the code queried the database.
Screenshot of the Market Model
Screenshot of the solution I found out that converting the market cap(market_cap) before adding it to the dictionary(price_map) returns the correct value. You can see it in line 138.
What could cause existent data in a record to be returned as null?
import logging
from flask_restful import Resource
from api.resources.util.date_util import pretty_date_to_epoch,
epoch_to_pretty_date
from common.decorators import log_exception
from db.models import db, Market
log = logging.getLogger(__name__)
def map_date_to_price():
buy_date_list = ["2015-01-01", "2015-02-01", "2015-03-01"]
sell_date_list = ["2014-12-19", "2014-01-10", "2015-01-20",
"2016-01-10"]
date_list = buy_date_list + sell_date_list
market_list = []
price_map = {}
for day in date_list:
market_list.append(db.session.query(Market).
filter(Market.pretty_date == day).first())
for market in market_list:
price_map[market.pretty_date] = market.market_cap
return price_map
The two fields that are (apparently) being retrieved as null are both db.Numeric. http://docs.sqlalchemy.org/en/latest/core/type_basics.html notes that these are, by default, backed up by a decimal.Decimal object, which I'll bet can't be converted to JSON, so what comes back form Market.__repr__() will show them as null.
I would try adding asdecimal=False to the two db.Numeric() calls in Market.
Related
I am trying to accomplish something that is pretty easy to do in SQL, but seemingly very challenging to do in SSIS without using SQL. Basically, I need to consolidate and concatenate a field of a many-to-one relationship.
Given entities: [Contract Item] (many) to (one) [Account]
There is a field [ari_productsummary] that contains the product listed on the Contract Item entity. We want to write that value to the Account as [ari_activecontractitems]. However, an Account may have more than one Contract Item record associated to it, in which case, we want to concatenate those values. We also only want the distinct values to be concatenated (distinct rows already solved within my data flow).
This can be accomplished by writing to a temporary table, and then using a query or view to obtain the summarized results as followed. I created a SQL table called TESTTABLE that contains the [ari_productsummary] from the Contract Item entity along with the referring [accountid] to map it back to Account. I then wrote the following query as a view:
SELECT distinct accountid,
(SELECT TT2.ari_productsummary + '; '
FROM TESTTABLE TT2
WHERE TT2.accountid = TT.accountid
FOR XML PATH ('')
) AS 'ari_activecontractitems'
FROM TESTTABLE TT
Executing that Query provides me the results that I want, which I can then use for importing into the Account entity as shown below:
But how do I do this in a SSIS dataflow without writing to a SQL table as a temporary placeholder for the data?? I want to do the entire process inside one dataflow container, without using a temporary SQL table/view. The whole summarization process needs to be done on the fly:
Does anyone have a solution that doesn't require a temporary SQL table/view/query, but is contained entirely within the data flow?
I am using VS 2017 and the KingswaySoft Dynamic CRM 365 ETL toolset to develop my solution/package.
Spit balling here as I don't Dynamics nor do I have the custom components.
Data Flow 1 - Contract aggregation
The purpose of this data flow is to replicate your logic in the elegant query you provided and shove that into a Cache Connection Manager (see Notes for 2008+ at the end)
KingswaySoft Dynamics Source -> Script Task -> Cache Transform
If you want to keep the sort in there, do it before the script task. The implementation I'll take with the Script Task is that it's fully blocking - that is all the rows must arrive before it can send any on. Tasks like the Merge Join are only partially blocking because the requirement of sorted data means that once you no longer have a match for the current item, you can send it on down the pipeline.
The Script Task is going to be asynchronous transformation. You'll have two output columns, your key accountid and your new derived column of ari_activecontractitems. That column will might need to be big - you'll know your data best but if it's a blob type in Dynamics (> 4k unicode or > 8k ascii characters) then you'll have to define the data type as DT_TEXT/DT_NTEXT
As inputs, you'll select accountid and ari_productsummary from your source.
The code should be pretty easy. We're going to accumulate the inbound data into a Dictionary.
// member variable
Dictionary<string, List<string>> accumulator;
The PreProcess method, we'll tack this in there to initialize our variable
// initialize in PreProcess method
accumulator = new Dictionary<string, List<string>>();
In the OnBufferRowSent (name approx)
// simulate the inbound queue
// row_id would be something like Rows.row_id
if (!accumulator.ContainsKey(row_id))
{
// Create an empty dictionary for our list
accumulator.Add(row_id, new List<string>());
}
// add it if we don't have it
if (!accumulator[row_id].Contains(invoice))
{
accumulator[row_id].Add(invoice);
}
Once you get the signal sent of no more data available, that's when you start buffering output data. The auto generated code will have placeholders for all this.
// This is how we shove data out the pipe
foreach(var kvp in accumulator)
{
// approximately thus
OutputBuffer1.AddRow();
OutputBuffer1.row_id = kvp.Key;
OutputBuffer1.ari_productsummary = string.Join("; ", kvp.Value);
}
We have an upcoming release that comes with a component that does exactly what you are trying to achieve without the need of writing custom code. The feature is currently under preview, please reach out to us for private access to the feature. You can find our contact information on our website.
UPDATE - June 5, 2020, we have made the components available for public access at https://www.kingswaysoft.com/products/ssis-productivity-pack/ as a result of our 2020 Release Wave 1. We have two components available that serve this kind of purpose. The Composition component will take input values and transform into a composite value in a SSIS column. The Decomposition component does the opposite, it would take an input value and split it into multiple rows using either delimiter-based text splitting or XML/JSON array splitting.
I have pulled in data from a number of csv files, as well as a database. I wish to use a merge function to make a dataframe isolating the phone numbers that are contained in both dataframes(one originating from csv, the other originating from the database). However, the dataframe from the database displays as type 'nonetype.' This disallows any operation such as merge. How can i change this to allow the operation?
The data comes in from the database as a list of tuples. I then convert this to a dataframe. However, as stated above, it displays as 'nonetype.' I'm assuming at the moment I am confused about about how dataframes handle data types.
#Grab Data
mycursor = mydb.cursor()
mycursor.execute("SELECT DISTINCT(Cell) FROM crm_data.ap_clients Order By Cell asc;")
apclients = mycursor.fetchall()
#Clean Phone Number Data
for index, row in data.iterrows():
data['phone_number'][index] = data['phone_number'][index][-10:]
for index, row in data2.iterrows():
data2['phone_number'][index] = data2['phone_number'][index][-10:]
for index, row in data3.iterrows():
data3['phone_number'][index] = data3['phone_number'][index][-10:]
#make data frame from csv files
fbl = pd.concat([data,data2,data3], axis=0, sort=False)
#make data frame from apclients(database extraction)
apc = pd.DataFrame(apclients)
#perfrom merge finding all records in both frames
successfulleads= pd.merge(fbl, apc, left_on ='phone_number', right_on='0')
#type(apc) returns NoneType
The expected results are to find all records in both dataframes, along with a count so that I may compare the two sets. Any help is greatly appreciated from this great community :)
So it looks like I had a function to rename the column of the dataframe as shown below:
apc = apc.rename(columns={'0': 'phone_number'}, inplace=True)
for col in apc.columns:
print(col)
the code snippet out of the above responsible:
inplace=True
This snippet dictates whether or not the object is modified in the dataframe, or whether a copy is made. The return type on said object is of nonetype.
Hope this helps whoever ends up in my position. A great thanks again to the community. :)
I need to export the result of a query of neo4j database to JSON or CSV, including relations and nodes, my query is this:
MATCH
(s:Socio)-[:ES_SOCIO_DE]->(p1:Empresas)-[:OFERTA_A]->(lic:Licitaciones)<-[:OFERTA_A]-(p2:Empresas)<-[:ES_SOCIO_DE]-(s:Socio)
WHERE ID(p1) <> ID(p2) RETURN * limit 100
but when I tried to export it to GraphML for example, it only exports the nodes
Image
Do you have access to the Neo4j browser interface for your installation? Usually, the URL will be something like:
http://[IP_ADDRESS_OF_YOUR_NEO4J_SERVER]:7474/browser/
In the browser interface, you can run your query in the query box, then click either the 'Text' or 'Table' panel on the left side of the returned query results box and you will see that you now have the option to 'Export CSV' in the top right portion of the returned query results box.
You can then either open the CSV directly or save it - and it will contain the nodes and the relationship properties.
If you want to return the type of the relationship (rather than just the properties) - which I have a hunch may be the case - return the relationship variable encapsulated in the built-in type() function. For example, using Neo4j's sample Movie database, I run the following query:
optional match (z:Person)-[x:ACTED_IN]->(v:Movie)
where z.name = "Tom Cruise"
return z,type(x),v
With the above query, rather than returning me the properties of his [:ACTED_IN] relationship, it will simply return "ACTED_IN"
Edit: Judging from your included image, which I admittedly did not notice initially, it looks like the relationships being returned are zero. Are you sure that the relationship that you are specifying actually exists?
I have successfully imported geodata (originally from a shapefile, converted to CSV) into my RavenDB. I am now trying to access the data with a naive, simplistic select (sanity check to see if everything's there) but I can't get any data member values back. Since I am a total RavenDB newbie and haven't created the data myself (programmatically), my approach was to define a class that has the same name as what I find in Raven Studio (minus the automatically-appended plural 's') under Raven-Entity-Name, and to declare each of the data members to be of type string.
The query runs through and retrieves the first 128 results, but all the data members are null. I used this:
List<AdministrativeArea> AdministrativeArea = session
.Query<AdministrativeArea>()
.ToList();
Looking at the entries in Raven Studio, I can see that some of the data member values of the documents are coloured blue (so are probably already type-cast to be integer) but that shouldn't be the cause of ALL the data members showing up as null...
No exceptions are being thrown, and the query list contains elements. What am I doing wrong here ?
Thanks for your help !
The problem was the instantiation of int data members. Even when declaring the int members to be nullable, the problem of empty strings came up and prevented correct instantiation of the objects.
I supose that when CSV imports are used, and in some of the cases the field/data member comes up "empty" (but as a string type), and in other cases, you DO have numbers, then you have to resort to declaring them all to be strings. The only other solution I could think of is to adapt the CSV import code, for which I am still too new to RavenDB to think about.
I'm having a strange issue.
I created a model observer for my user model. The model observer is being run at 'saving'. when I dump the object at the very end of the user model to be displayed ( this is just before it saves.. according to laravel docs ) it displays all the attributes set correctly for the object, I've even seen an error that showed the correct attributes as set and being inserted into my database table. However, after the save has been completed and I query the database, two of the fields are not saved into the table.
There is no code written by myself sitting between the point where I dumped the attributes to check that they had been set and the save operation to the database. so I have no idea what could be causing this to happen. All the names are set correctly, and like I said, the attributes show as being inserted into the database, they just never end up being saved, I receive no error messages and only two out of ten attributes aren't being saved.
In my searches I have found many posts detailing that the $fillable property should be set, or issues relating to a problem with variables being misnamed or unset, however because I already have the specific attributes not being saved specified in the $fillable array, on top of the fact that they print out exactly as expected pre save, I don't believe those issues are related to the problem I am experiencing.
to save I'm calling:
User::create(Input::all());
and then the observer that handles the data looks like this:
class UserObserver {
# a common key between the city and state tables, helps to identify correct city
$statefp = State::where('id',$user->state_id)->pluck('statefp');
# trailing zeros is a function that takes the first parameter and adds zeros to make sure
# that in this case for example, the dates will be two characters with a trailing zero,
# based on the number specified in the second parameter
$user->birth_date = $user->year.'-'.$user->trailingZeros( $user->month, 2 ).'-'.$user->trailingZeros( $user->day, 2 );
if(empty($user->city)){
$user->city_id = $user->defaultCity;
}
$user->city_id = City::where( 'statefp', $statefp )->where('name', ucfirst($user->city_id))->pluck('id');
# if the user input zip code is different then suggested zip code then find location data
# on the input zip code input by the user from the geocodes table
if( $user->zip !== $user->defaultZip ){
$latlon = Geocode::where('zip', $user->zip)->first();
$user->latitude = $latlon['latitude'];
$user->longitude = $latlon['longitude'];
}
unset($user->day);
unset($user->month);
unset($user->year);
unset($user->defaultZip);
unset($user->defaultCity);
}
that is the code for the two values that aren't being set, when I run
dd($user);
all the variables are set correctly, and show up in the mysql insert attempt screen with correct values, but they do not persist past that point.. it seems to me that possibly mysql is rejecting the values for the city_id and the birth_date. However, I cannot understand why, or whether it is a problem with Laravel or mysql.
since I was calling
User::create();
I figured I'd try to have my observer listen to:
creating();
I'm not sure why it only effected the date and city variables, but changing the function to listen at creating() instead of saving() seems to have solved my problem.