I want to save the encoded part just before decoder stage in autoencoder model of Keras with Tensorflow backend.
For instance;
encoding_dim = 210
input_img = Input(shape=(5184,))
encoded = Dense(2592, activation='relu')(input_img)
encoded1 = Dense(encoding_dim, activation='relu')(encoded)
decoded = Dense(encoding_dim, activation='relu')(encoded1)
decoded = Dense(5184, activation='sigmoid')(decoded)
I want to save the encoded1 as csv file after the autoencoder training. Suppose the output of Dense will be (nb_samples, output_dim).
Thank you
Try:
autoencoder = Model(input_img, decoded)
encoder = Model(input_img, encoded1)
autoencoder.compile(loss=my_loss, optimizer=my_optimizer)
autoencoder.fit(my_data, my_data, .. rest of fit params)
numpy.savetxt("encoded1.csv", encoder.predict(x), delimiter=",")
Moreover - I don't know what kind of data do you have but I advise you to use linear activation is a last layer and mse loss function.
Related
I am having a pretrained model in json format but i want to use it in python app, is there a way of converting this model to keras h5 Model format?
As per https://www.tensorflow.org/api_docs/python/tf/keras/models/, it should look something like:
json_file_path = "/path/to/json"
with open(json_file_path, "r") as json_file:
json_savedModel= json_file.read()
model_json = tf.keras.models.model_from_json(json_savedModel)
model_json.save("model_name.h5", save_format="hf")
Edit: And then to load the h5 model:
h5_file = "/path/to/model_name.h5"
model_h5 = tf.keras.models.load_model(h5_file)
I'm working on a project and need to parse a GeoJSON shapefile of flight route airspace in the US. The data is coming from the FAA open data portal: https://adds-faa.opendata.arcgis.com/datasets/faa::route-airspace/about
There seems to be some relevant documentation at /workspace/documentation/product/geospatial-docs/vector_data_in_transforms where it mentions:
A typical Foundry Ontology pipeline for geospatial vector data may include the following steps:
- Convert into rows with GeoJSON representation of the shape for each feature.
However there isn't actually any guidance on how to go about doing this when the source is a single GeoJSON file with a FeatureCollection and the desired output is a dataset with one row per Feature in the collection.
Anyone have a code snippet for accomplishing this? Seems like a pretty generic task in Foundry.
I typically do something like this:
import json
with open('Route_Airspace.geojson', 'r') as f:
data = json.load(f)
rows = []
for feature in data['features']:
row = {
'geometry': json.dumps(feature['geometry']),
'properties': json.dumps(feature['properties']),
'id': feature['properties']['OBJECTID']
}
rows.append(row)
Note you can leave out the properties, but I like to keep them in this step in case I need them later. Also note this is a good place to set each row's primary key as well (the Features in this dataset have an OBJECTID property, but this may vary)
This rows list can then be used to initialise a Pandas dataframe:
import pandas as pd
df = pd.DataFrame(rows)
or a Spark dataframe (*assuming you're doing this within a transform):
df = ctx.spark_session.createDataFrame(rows)
The resulting dataframes will have one row per Feature, where that feature's shape is contained within the geometry column.
Full example within transform:
from transforms.api import transform, Input, Output
import json
#transform(
out=Output('Path/to/output'),
source_df=Input('Path/to/source'),
)
def compute(source_df, out, ctx)
with source_df.filesystem().open('Route_Airspace.geojson', 'r') as f:
data = json.load(f)
rows = []
for feature in data['features']:
row = {
'geometry': json.dumps(feature['geometry']),
'properties': json.dumps(feature['properties']),
'id': feature['properties']['OBJECTID']
}
rows.append(row)
df = ctx.spark_session.createDataFrame(rows)
out.write_dataframe(df)
Note that for this to work your GeoJSON file needs to be uploaded into a "dataset without a schema" so the raw file becomes accessible via the FileSystem API.
i'm using micropython in the newest version. I also us an DS18b20 temperature sensor. An adress of theses sensor e.g. is "b'(b\xe5V\xb5\x01<:'". These is the string representation of an an bytearray. If i use this to save the adress in a json file, i run in some problems:
If i store directly "b'(b\xe5V\xb5\x01<:'" after reading the json-file there are no single backslahes, and i get b'(bxe5Vxb5x01<:' inside python
If i escape the backslashes like "b'(b\xe5V\xb5\x01<:'" i get double backslashes in python: b'(b\xe5V\xb5\x01<:'
How do i get an single backslash?
Thank you
You can't save bytes in JSON with micropython. As far as JSON is concerned that's just some string. Even if you got it to give you what you think you want (ie. single backslashes) it still wouldn't be bytes. So, you are faced with making some form of conversion, no-matter-what.
One idea is that you could convert it to an int, and then convert it back when you open it. Below is a simple example. Of course you don't have to have a class and staticmethods to do this. It just seemed like a good way to wrap it all into one, and not even need an instance of it hanging around. You can dump the entire class in some other file, import it in the necessary file, and just call it's methods as you need them.
import math, ujson, utime
class JSON(object):
#staticmethod
def convert(data:dict, convert_keys=None) -> dict:
if isinstance(convert_keys, (tuple, list)):
for key in convert_keys:
if isinstance(data[key], (bytes, bytearray)):
data[key] = int.from_bytes(data[key], 'big')
elif isinstance(data[key], int):
data[key] = data[key].to_bytes(1 if not data[key]else int(math.log(data[key], 256)) + 1, 'big')
return data
#staticmethod
def save(filename:str, data:dict, convert_keys=None) -> None:
#dump doesn't seem to like working directly with open
with open(filename, 'w') as doc:
ujson.dump(JSON.convert(data, convert_keys), doc)
#staticmethod
def open(filename:str, convert_keys=None) -> dict:
return JSON.convert(ujson.load(open(filename, 'r')), convert_keys)
#example with both styles of bytes for the sake of being thorough
json_data = dict(address=bytearray(b'\xFF\xEE\xDD\xCC'), data=b'\x00\x01\02\x03', date=utime.mktime(utime.localtime()))
keys = ['address', 'data'] #list of keys to convert to int/bytes
JSON.save('test.json', json_data, keys)
json_data = JSON.open('test.json', keys)
print(json_data) #{'date': 1621035727, 'data': b'\x00\x01\x02\x03', 'address': b'\xff\xee\xdd\xcc'}
You may also want to note that with this method you never actually touch any JSON. You put in a dict, you get out a dict. All the JSON is managed "behind the scenes". Regardless of all of this, I would say using struct would be a better option. You said JSON though so, my answer is about JSON.
I follow Object Detection Demo in https://github.com/tensorflow/models/blob/master/research/object_detection/object_detection_tutorial.ipynb:
# Actual detection.
output_dict = run_inference_for_single_image(image_np, detection_graph)
But I want to convert output_dict (output from function run_inference_for_single_image(image_np, detection_graph)) to COCO annotation JSON type so I can input it to make benchmark between different Object Detection models.
Here is code to benchmark model: https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoEvalDemo.ipynb
#initialize COCO detections api
resFile='%s/results/%s_%s_fake%s100_results.json'
resFile = resFile%(dataDir, prefix, dataType, annType)
cocoDt=cocoGt.loadRes(resFile)
But you need to input a COCO Json type.
Are there anyone can tell me how to Convert from output_dict to COCO Json?
This is what you are looking for. The last code snippet:
https://lijiancheng0614.github.io/2017/08/22/2017_08_22_TensorFlow-Object-Detection-API/
I am a newbie and have this Keras UNet model that I need to train.
https://github.com/Swapneel7/Deep-Learning-Keras/blob/master/UNet%20Keras.pdf
The input tensor does follow the required dimensions of 1024*1024*1.
Input:-
Parse the mat into array
import scipy.io as sio
import tensorflow as tf
import numpy as np
input1=sio.loadmat('D:\\Users\\svekhande\\tempx.mat')
TensorInput1=input1['temp']
TensorInput= np.expand_dims(TensorInput1,2)
print(TensorInput.shape)
Model:-
def unet(input_size=(1024,1024,1)):
inputs = Input(input_size)
...
.
Training:-
model.fit(TensorInput, steps_per_epoch=1, epochs=3)
Error
IndexError: list index out of range
The error indicates I exceeded the bound so I tried feeding smaller tensors (1023*1023*1), changing the network shape and resetting the graph but it did not work hence posting.