JINA#4428[C]:Can not fetch the URL of Hubble from `api.jina.ai` - json

I was trying out the Semantic Wikipedia Search from jina-ai.
This is the error I am getting after running python app.py -t index.
app.py is used to index the data.
JINA#4489[C]:Can not fetch the URL of Hubble from api.jina.ai
HubIO#4489[E]:Error while pulling jinahub+docker://TransformerTorchEncoder:
JSONDecodeError('Expecting value: line 1 column 1 (char 0)')
This is app.py:
__copyright__ = "Copyright (c) 2021 Jina AI Limited. All rights reserved."
__license__ = "Apache-2.0"
import os
import sys
import click
import random
from jina import Flow, Document, DocumentArray
from jina.logging.predefined import default_logger as logger
MAX_DOCS = int(os.environ.get('JINA_MAX_DOCS', 10000))
def config(dataset: str):
if dataset == 'toy':
os.environ['JINA_DATA_FILE'] = os.environ.get('JINA_DATA_FILE', 'data/toy-input.txt')
elif dataset == 'full':
os.environ['JINA_DATA_FILE'] = os.environ.get('JINA_DATA_FILE', 'data/input.txt')
os.environ['JINA_PORT'] = os.environ.get('JINA_PORT', str(45678))
cur_dir = os.path.dirname(os.path.abspath(__file__))
os.environ.setdefault('JINA_WORKSPACE', os.path.join(cur_dir, 'workspace'))
os.environ.setdefault('JINA_WORKSPACE_MOUNT',
f'{os.environ.get("JINA_WORKSPACE")}:/workspace/workspace')
def print_topk(resp, sentence):
for doc in resp.data.docs:
print(f"\n\n\nTa-Dah🔮, here's what we found for: {sentence}")
for idx, match in enumerate(doc.matches):
score = match.scores['cosine'].value
print(f'> {idx:>2d}({score:.2f}). {match.text}')
def input_generator(num_docs: int, file_path: str):
with open(file_path) as file:
lines = file.readlines()
num_lines = len(lines)
random.shuffle(lines)
for i in range(min(num_docs, num_lines)):
yield Document(text=lines[i])
def index(num_docs):
flow = Flow().load_config('flows/flow.yml')
data_path = os.path.join(os.path.dirname(__file__), os.environ.get('JINA_DATA_FILE', None))
with flow:
flow.post(on='/index', inputs=input_generator(num_docs, data_path),
show_progress=True)
def query(top_k):
flow = Flow().load_config('flows/flow.yml')
with flow:
text = input('Please type a sentence: ')
doc = Document(content=text)
result = flow.post(on='/search', inputs=DocumentArray([doc]),
parameters={'top_k': top_k},
line_format='text',
return_results=True,
)
print_topk(result[0], text)
#click.command()
#click.option(
'--task',
'-t',
type=click.Choice(['index', 'query'], case_sensitive=False),
)
#click.option('--num_docs', '-n', default=MAX_DOCS)
#click.option('--top_k', '-k', default=5)
#click.option('--dataset', '-d', type=click.Choice(['toy', 'full']), default='toy')
def main(task, num_docs, top_k, dataset):
config(dataset)
if task == 'index':
if os.path.exists(os.environ.get("JINA_WORKSPACE")):
logger.error(f'\n +---------------------------------------------------------------------------------+ \
\n | 🤖🤖🤖 | \
\n | The directory {os.environ.get("JINA_WORKSPACE")} already exists. Please remove it before indexing again. | \
\n | 🤖🤖🤖 | \
\n +---------------------------------------------------------------------------------+')
sys.exit(1)
index(num_docs)
elif task == 'query':
query(top_k)
if __name__ == '__main__':
main()
This is flow.yml
version: '1' # This is the yml file version
with: # Additional arguments for the flow
workspace: $JINA_WORKSPACE # Workspace folder path
port_expose: $JINA_PORT # Network Port for the flow
executors: # Now, define the executors that are run on this flow
- name: transformer # This executor computes an embedding based on the input text documents
uses: 'jinahub+docker://TransformerTorchEncoder' # We use a Transformer Torch Encoder from the hub as a docker container
- name: indexer # Now, index the text documents with the embeddings
uses: 'jinahub://SimpleIndexer' # We use the SimpleIndexer for this purpose
And when I try to execute app.py -t index
This is the error:
JINA#3803[C]:Can not fetch the URL of Hubble from `api.jina.ai` HubIO#3803[E]:Error while pulling jinahub+docker://TransformerTorchEncoder: JSONDecodeError('Expecting value: line 1 column 1 (char 0)')

I think this just happened because the API was down. It should work now.

Related

Special Characters and Converting Problems Using Tabula for PDF to Proper CSV

The code:
#Import the required Module
import tabula
# Read a PDF File
df=tabula.read_pdf("C:/Users/Desktop/abstract/abstract.pdf",encoding='cp1252', pages='all')
#Total page number can change. All pages must be taken. (to be generic)
# convert PDF into CSV
df1=df.to_csv('C:/Users/Desktop/abstract.pdf')
print(df1)
Hello friends, I have a monthly account statement in pdf. I want to get the name and period information as text, save the date, info, amount, and gift information as CSV and read it.
I tried something called tabula read but couldn't get a file the way I wanted. In addition, there are special characters in the pdf file. These are also decoded incorrectly (ğ, ü, ç, etc.)
How can I get the format which I want? So mean I can't reach İşlem tarihi,Açıklama,Tutar,Bankomat Para columns in CSV file or Is there any better way to convert pdf to CSV file?
original version (2 pages):
original version end of the page:
The code:
#libraries
import pandas as pd
import fitz
import io
def set_texts(pdf_files:list):
print("starting to text process")
#This function reads pdf and gets "CRC-32" components as texts
for pdf_file in pdf_files:
with fitz.open(pdf_file) as doc:
text = ""
for page in doc:
new_text = page.get_text()
text += new_text
return text
file_names = [r"C:/Users/Desktop/abstract/Alışveris_Özeti.pdf"]
text = set_texts(file_names)
buffer = io.StringIO(text)
new_text = ""
flag = False
i = 0
for line in buffer.readlines():
# print(line)
if "Bankomat Para Bilgileriniz" in line:
flag = False
elif "Bankomat Para (TL)" in line:
flag = True
elif "Vakıfbank" in line:
flag = False
elif flag:
new_text += line
elif "Sayın" in line :
name=(line.lstrip("Sayın ")).replace(",","")
print(name)
buffer = io.StringIO(new_text)
text_list = buffer.readlines()
# /n correction
a = list(map(lambda x: x.replace('\n', ''), text_list))
# Converting 4 spaces to single space
b = list(map(lambda x: x.replace(' ', ' '), a))
# card vocabulay
c = list(map(lambda x: x.replace('BANKOMAT KART ', 'BANKOMAT KART'), b))
# undesired words
stopwords = ['BANKOMAT KART','İŞLEMİ', 'ALIŞVERİŞ EKSTRESİ', 'Dekont yerine kullanılmaz. Uyuşmazlık halinde Banka kayıtları esas alınacaktır', 'www.vakifbank.com.tr I 0850 222 0 724', 'Türkiye Vakıflar Bankası T.A.O. Büyük Mükellefler V.D. 9220034970 Sicil Numarası: 776444','Saray Mahallesi Dr. Adnan Büyükdeniz Caddesi No :7 / A-B Ümraniye /İSTANBUL Mersis: 0922003497000017','Saray Mahallesi Dr. Adnan Büyükdeniz Caddesi No :7 / A-B Ümraniye /İSTANBUL Mersis: 0922003497000017 Sf 2 \\ 3 ']
d = list(filter(lambda w: w not in stopwords, c))
e = list(map(lambda x: x.replace('CÜZDANDAN HESABA TRANSFER ', 'CÜZDANDAN HESABA TRANSFER İŞLEMİ'), d))
# Align the list elements by 4 in the appropriate order according to the 4 columns of the df
z=[]
for i in range(int(len(e)/4)):
y=((e[i*4:i*4+4]))
z.append(y)
df = pd.DataFrame( z,columns=['ISLEM TARIHI', 'ACIKLAMA','TUTAR', "BANKOMAT PARA"])
# creating csv file
is_file=os.path.isfile('C:/Users/Desktop/abstract/Alışveris_Özeti.csv')
if is_file==False:
df.to_csv("Alışveris_Özeti.csv", index=False)
print('CSV file has created...')
else:
print("CSV file already exists.")

Python script that can auto-annotate the images

I am using the https://github.com/mdhmz1/Auto-Annotate repo. I have tried to custom train my own dataset which has it own COCO JSON format file.
When I try to run
python3 customTrain.py train --dataset=path/to/dir --weights=coco
I get the following error:
Traceback (most recent call last):
File "customTrain.py", line 279, in
train(model)
File "customTrain.py", line 179, in train
dataset_train.load_custom(args.dataset, "train")
File "customTrain.py", line 87, in load_custom
annotations = [a for a in annotations if a['regions']]
File "customTrain.py", line 87, in
annotations = [a for a in annotations if a['regions']]
TypeError: list indices must be integers or slices, not str
My customtrain.py looks like the following:
import os
import sys
import json
import datetime
import numpy as np
import skimage.draw
Root directory of the project
ROOT_DIR = "/home/hiwi/Auto-Annotate"
Import Mask RCNN
sys.path.append(ROOT_DIR) # To find local version of the library
from mrcnn.config import Config
from mrcnn import model as modellib, utils
Path to trained weights file
COCO_WEIGHTS_PATH = os.path.join(ROOT_DIR, "mask_rcnn_coco.h5")
Directory to save logs and model checkpoints, if not provided
through the command line argument --logs
DEFAULT_LOGS_DIR = os.path.join(ROOT_DIR, "logs")
############################################################
Configurations
############################################################
class CustomConfig(Config):
"""Configuration for training on the toy dataset.
Derives from the base Config class and overrides some values.
"""
# Give the configuration a recognizable name
NAME = "custom"
IMAGES_PER_GPU = 1
# Number of classes (including background)
NUM_CLASSES = 1 + 2 # Background + 2 classes
# Number of training steps per epoch
STEPS_PER_EPOCH = 100
# Skip detections with < 90% confidence
DETECTION_MIN_CONFIDENCE = 0.9
############################################################
Dataset
############################################################
class CustomDataset(utils.Dataset):
def load_custom(self, dataset_dir, subset):
"""Load a subset of the Custom dataset.
dataset_dir: Root directory of the dataset.
subset: Subset to load: train or val
"""
# Add classes. We have only one class to add.
self.add_class("custom", 0, "Primary_Track")
self.add_class("custom", 1, "Secondary_Track")
# Train or validation dataset?
assert subset in ["train", "val"]
dataset_dir = os.path.join(dataset_dir, subset)
# Load annotations
# VGG Image Annotator (up to version 1.6) saves each image in the form:
# { 'filename': '28503151_5b5b7ec140_b.jpg',
# 'regions': {
# '0': {
# 'region_attributes': {},
# 'shape_attributes': {
# 'all_points_x': [...],
# 'all_points_y': [...],
# 'name': 'polygon'}},
# ... more regions ...
# },
# 'size': 100202
# }
# We mostly care about the x and y coordinates of each region
# Note: In VIA 2.0, regions was changed from a dict to a list.
annotations1 = json.load(open(os.path.join(dataset_dir, "train.json")))
annotations = list(annotations1.values()) # don't need the dict keys
# The VIA tool saves images in the JSON even if they don't have any
# annotations. Skip unannotated images.
annotations = [a for a in annotations if a['regions']]
# Add images
for a in annotations:
# Get the x, y coordinaets of points of the polygons that make up
# the outline of each object instance. These are stores in the
# shape_attributes (see json format above)
# The if condition is needed to support VIA versions 1.x and 2.x.
if type(a['regions']) is dict:
polygons = [r['shape_attributes'] for r in a['regions'].values()]
else:
polygons = [r['shape_attributes'] for r in a['regions']]
#labelling each class in the given image to a number
custom = [s['region_attributes'] for s in a['regions']]
num_ids=[]
#Add the classes according to the requirement
for n in custom:
try:
if n['name']=="Primary_Track":
num_ids.append(0)
elif n['name']=='Secondary_Track':
num_ids.append(1)
except:
pass
# load_mask() needs the image size to convert polygons to masks.
# Unfortunately, VIA doesn't include it in JSON, so we must read
# the image. This is only managable since the dataset is tiny.
image_path = os.path.join(dataset_dir, a['filename'])
image = skimage.io.imread(image_path)
height, width = image.shape[:2]
self.add_image(
"custom",
image_id=a['filename'], # use file name as a unique image id
path=image_path,
width=width, height=height,
polygons=polygons,
num_ids=num_ids)
def load_mask(self, image_id):
"""Generate instance masks for an image.
Returns:
masks: A bool array of shape [height, width, instance count] with
one mask per instance.
class_ids: a 1D array of class IDs of the instance masks.
"""
# If not a custom dataset image, delegate to parent class.
image_info = self.image_info[image_id]
if image_info["source"] != "custom":
return super(self.__class__, self).load_mask(image_id)
num_ids = image_info['num_ids']
#print("Here is the numID",num_ids)
# Convert polygons to a bitmap mask of shape
# [height, width, instance_count]
info = self.image_info[image_id]
mask = np.zeros([info["height"], info["width"], len(info["polygons"])],
dtype=np.uint8)
for i, p in enumerate(info["polygons"]):
if p['name'] == 'polygon':
# Get indexes of pixels inside the polygon and set them to 1
rr, cc = skimage.draw.polygon(p['all_points_y'], p['all_points_x'])
else:
rr, cc = skimage.draw.rectangle((p['y'], p['x']), extent=(p['height'], p['width']))
rr[rr > mask.shape[0]-1] = mask.shape[0]-1
cc[cc > mask.shape[1]-1] = mask.shape[1]-1
mask[rr, cc, i] = 1
# Return mask, and array of class IDs of each instance. Since we have
# one class ID only, we return an array of 1s
num_ids = np.array(num_ids, dtype=np.int32)
return mask.astype(np.bool), num_ids.astype(np.bool), np.ones([mask.shape[-1]], dtype=np.int32)
#return mask.astype(np.bool), np.ones([mask.shape[-1]], dtype=np.int32)
def image_reference(self, image_id):
"""Return the path of the image."""
info = self.image_info[image_id]
if info["source"] == "Railtrack":
return info["path"]
else:
super(self.__class__, self).image_reference(image_id)
def train(model):
"""Train the model."""
# Training dataset.
dataset_train = CustomDataset()
dataset_train.load_custom(args.dataset, "train")
dataset_train.prepare()
# Validation dataset
dataset_val = CustomDataset()
dataset_val.load_custom(args.dataset, "val")
dataset_val.prepare()
# *** This training schedule is an example. Update to your needs ***
# Since we're using a very small dataset, and starting from
# COCO trained weights, we don't need to train too long. Also,
# no need to train all layers, just the heads should do it.
print("Training network heads")
model.train(dataset_train, dataset_val,
learning_rate=config.LEARNING_RATE,
epochs=30,
layers='heads')
############################################################
Training
############################################################
if name == 'main':
import argparse
# Parse command line arguments
parser = argparse.ArgumentParser(
description='Train Mask R-CNN to detect custom objects.')
parser.add_argument("command",
metavar="<command>",
help="'train' or 'splash'")
parser.add_argument('--dataset', required=False,
metavar="/path/to/custom/dataset/",
help='Directory of the Custom dataset')
parser.add_argument('--weights', required=True,
metavar="/path/to/weights.h5",
help="Path to weights .h5 file or 'coco'")
parser.add_argument('--logs', required=False,
default=DEFAULT_LOGS_DIR,
metavar="/path/to/logs/",
help='Logs and checkpoints directory (default=logs/)')
parser.add_argument('--image', required=False,
metavar="path or URL to image",
help='Image to apply the color splash effect on')
parser.add_argument('--video', required=False,
metavar="path or URL to video",
help='Video to apply the color splash effect on')
args = parser.parse_args()
# Validate arguments
if args.command == "train":
assert args.dataset, "Argument --dataset is required for training"
elif args.command == "splash":
assert args.image or args.video,\
"Provide --image or --video to apply color splash"
print("Weights: ", args.weights)
print("Dataset: ", args.dataset)
print("Logs: ", args.logs)
# Configurations
if args.command == "train":
config = CustomConfig()
# Create model
if args.command == "train":
model = modellib.MaskRCNN(mode="training", config=config,
model_dir=args.logs)
# Select weights file to load
if args.weights.lower() == "coco":
weights_path = COCO_WEIGHTS_PATH
# Download weights file
if not os.path.exists(weights_path):
utils.download_trained_weights(weights_path)
elif args.weights.lower() == "last":
# Find last trained weights
weights_path = model.find_last()
elif args.weights.lower() == "imagenet":
# Start from ImageNet trained weights
weights_path = model.get_imagenet_weights()
else:
weights_path = args.weights
# Load weights
print("Loading weights ", weights_path)
if args.weights.lower() == "coco":
# Exclude the last layers because they require a matching
# number of classes
model.load_weights(weights_path, by_name=True, exclude=[
"mrcnn_class_logits", "mrcnn_bbox_fc",
"mrcnn_bbox", "mrcnn_mask"])
else:
model.load_weights(weights_path, by_name=True)
# Train or evaluate
if args.command == "train":
train(model)
else:
print("'{}' is not recognized. "
"Use 'train' or 'splash'".format(args.command))

How to handle newlines when loading a CSV into Apache Beam?

I am running into an issue where in some of my fields, there are new lines within the text.
My current code is as follows:
# Python's regular expression library
import re
import sys
# Beam and interactive Beam imports
import apache_beam as beam
from apache_beam.runners.interactive.interactive_runner import InteractiveRunner
import apache_beam.runners.interactive.interactive_beam as ib
p = beam.Pipeline(InteractiveRunner())
def print_row(element):
print(element)
def parse_file(element):
for line in csv.reader([element], quotechar='"', delimiter=',', lineterminator='\n', quoting=csv.QUOTE_ALL, skipinitialspace=True):
return line
parsed_csv = p | 'Read input file' >> beam.io.ReadFromText("gs://ny-data/AB_NYC_2019.csv")| 'Parse file' >> beam.Map(parse_file)
split = parsed_csv | beam.Map(lambda x: x[0]) | beam.Map(print)
p.run()
I am running into issues because some of the text appears as so:
The BLUE OWL:
VEGETARIAN WBURG W PATIO & BACKYARD!
Any thoughts on how to proceed?
ReadFromText reads inputs one line at a time. As suggested before, you can use the Dataframe read_csv, or you could create a PCollection of paths and open/read them in a DoFn.
For example, you could write
def read_csv_file(file_metadata):
with beam.io.filesystems.FileSystems.open(file_metadata.path) as fin:
for row in csv.reader(fin):
yield row
rows = (
p
| beam.io.fileio.MatchFiles('/pattern/to/files/*.csv') # emits FileMetadatas
| beam.FlatMap(read_csv_file)) # emits rows

Zabbix Web scenarios variables random number or other function

I need post variable with random number value. How can i generate random variable in web scenario? Can i run some script or macro to generate random value for scenario or step?
There is no native way to do it, as you guessed you can make it work with a macro and a custom script.
You can define a {$RANDOM} host macro and use it in the web scenario step as a post field value.
Then you have to change it periodically with a crontabbed script, a python sample like:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
Set a random macro to a value.
Provide user from the commandline or from Env var support:
# export ZABBIX_SERVER='https://your_zabbix_host/zabbix/'
# export ZABBIX_USERNAME='admin'
# export ZABBIX_PASSWORD='secretPassword'
$ ./setRandomMacro.py -u admin -p zabbix -Z http://yourzabbix -H yourHost -M '{$RANDOM}'
Connecting to http://yourzabbix
Host yourHost (Id: ----)
{$RANDOM}: current value "17" -> new value "356"
$ ./setRandomMacro.py -u admin -p zabbix -Z http://yourzabbix -H yourHost -M '{$RANDOM}'
Connecting to http://yourzabbix
Host yourHost (Id: ----)
{$RANDOM}: current value "356" -> new value "72"
"""
from zabbix.api import ZabbixAPI
import json
import argparse
import getopt
import sys
import os
import random
# Class for argparse env variable support
class EnvDefault(argparse.Action):
# From https://stackoverflow.com/questions/10551117/
def __init__(self, envvar, required=True, default=None, **kwargs):
if not default and envvar:
if envvar in os.environ:
default = os.environ[envvar]
if required and default:
required = False
super(EnvDefault, self).__init__(default=default, required=required,
**kwargs)
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest, values)
def jsonPrint(jsonUgly):
print(json.dumps(jsonUgly, indent=4, separators=(',', ': ')))
def ArgumentParser():
parser = argparse.ArgumentParser()
parser.add_argument('-Z',
required=True,
action=EnvDefault,
envvar='ZABBIX_SERVER',
help="Specify the zabbix server URL ie: http://yourserver/zabbix/ (ZABBIX_SERVER environment variable)",
metavar='zabbix-server-url')
parser.add_argument('-u',
required=True,
action=EnvDefault,
envvar='ZABBIX_USERNAME',
help="Specify the zabbix username (ZABBIX_USERNAME environment variable)",
metavar='Username')
parser.add_argument('-p',
required=True,
action=EnvDefault,
envvar='ZABBIX_PASSWORD',
help="Specify the zabbix username (ZABBIX_PASSWORD environment variable)",
metavar='Password')
parser.add_argument('-H',
required=True,
help="Hostname",
metavar='hostname')
parser.add_argument('-M',
required=True,
help="Macro to set",
metavar='macro')
return parser.parse_args()
def main(argv):
# Parse arguments and build work variables
args = ArgumentParser()
zabbixURL = args.Z
zabbixUsername = args.u
zabbixPassword = args.p
hostName = args.H
macroName = args.M
# API Connect
print('Connecting to {}'.format(zabbixURL))
zapi = ZabbixAPI(url=zabbixURL, user=zabbixUsername,
password=zabbixPassword)
hostObj = zapi.host.get(search={'host': hostName}, output='hostids')
print('Host {} (Id: {})'.format(hostName, hostObj[0]['hostid']))
currentMacro = zapi.usermacro.get(
hostids=hostObj[0]['hostid'], filter={'macro': macroName})
if (currentMacro):
newMacroValue = random.randint(1, 1001)
print('{}: current value "{}" -> new value "{}"'.format(macroName,
currentMacro[0]['value'], newMacroValue))
zapi.usermacro.update(
hostmacroid=currentMacro[0]['hostmacroid'], value=newMacroValue)
else:
print('No {} macro found on host {}'.format(macroName, hostName))
if __name__ == "__main__":
main(sys.argv[1:])

Passenger running Flask site returning 500 error on Dreamhost

I am having trouble getting Passenger to work with Flask. I am running the Flask site on DreamHost inside a virutalenv running Python 3.5.2. The application should receive input from a query string in the URL, parse a server-side JSON file, search the JSON file for said query string, reformat the JSON, and send it back to the client. Instead, it gives me the 500 error code. Here is my code. Please pardon all the logging function calls. I've done what I can by searching around.
import logging
logging.basicConfig(filename='wsgi_flask.log', level=logging.DEBUG)
try:
import sys, os
logging.info('line 7')
VENV = os.path.join(os.environ['HOME'], '.pyenv', 'versions', '3.5.2', 'envs', 'flask')
INTERP = os.path.join(VENV, 'bin', 'python')
logging.info('line 12')
if sys.executable != INTERP:
os.execl(INTERP, INTERP, *sys.argv)
logging.info('line 17')
sys.path.append(VENV)
sys.path.append(os.path.join(VENV, 'lib', 'python3.5', 'site-packages'))
sys.path.append(os.path.join(os.environ['HOME'], 'api.example.com'))
logging.info('line 23')
# INTERP = os.path.join(os.environ['HOME'], '.pyenv', 'versions', 'flask', 'bin', 'python')
# if sys.executable != INTERP:
# os.execl(INTERP, INTERP, *sys.argv)
# sys.path.append(os.getcwd())
from paste.exceptions.errormiddleware import ErrorMiddleware
logging.info('line 31')
# import argparse
import json
logging.info('line 36')
from flask import (Flask, abort, g, jsonify, make_response, redirect,
render_template, render_template_string, request, url_for, Response)
logging.info('line 41')
# parser = argparse.ArgumentParser()
# parser.add_argument('-d', '--debug', action='store_true')
# apargs = parser.parse_args()
# def debugLog(*args):
# if apargs.debug:
# for i in args:
# print(i)
application = Flask(__name__)
logging.info('line 54')
#application.route('/')
def index():
logging.info('line 58')
# if apargs.debug:
# data = json.load(open('/path/to/test.json', 'rb'))
# else:
logging.info('line 62')
try:
logging.info('line 64')
data = json.load(open('/path/to/test.json', 'rb'))
logging.info('line 66')
except:
logging.info('line 68')
logging.warning('Is it the JSON?')
raise('Error whilst loading JSON')
logging.info('line 71')
getForm = request.args.get('search') or ''
logging.info('line 73')
# debugLog(data, getForm)
jsonData = ""
logging.info('line 78')
for d in data:
logging.info('line 80 (for loop)')
# debugLog(d)
if getForm in d['text']:
logging.info('line 83 (for loop)')
jsonData = jsonData + str(d) + ','
logging.info('line 85 (for loop)')
logging.info('line 86 (for loop)')
logging.info('line 87 (out of loop)')
# debugLog(jsonData)
jsonData = str(jsonData)
logging.info('line 91')
jsonData = jsonData[:-1]
logging.info('line 93')
jsonData = jsonData.replace("'", '"')
logging.info('line 95')
jsonData = '[' + jsonData + ']'
logging.info('line 97')
# debugLog('\n', jsonData)
return Response(jsonify(json.loads(jsonData)), mimetype='application/json')
logging.info('line 101 (this should not show)')
#application.errorhandler(500)
def error(e):
logging.warning(e + '\n\n\n What\'s happening here')
return str(e)
logging.info('line 107')
application = ErrorMiddleware(application, debug=True, error_log='tmp/wsgi_error.log')
logging.info('line 109')
# if apargs.debug:
# application.run(host='0.0.0.0', debug=True, port=80)
except:
logging.warning('line 114 (beginning of outer most try-except)')
import sys, os
logging.warning('line 116')
from traceback import format_list, extract_tb
logging.warning('line 119')
(extype, value, trace) = sys.exc_info()
logging.warning('line 121')
sys.stderr.buffer.write("%s:%s\n%s" % (extype, value,''.join(format_list(extract_tb(trace)))))
logging.warning('line 124 (EOF)')
This is my directory tree:
.
|-- .python-version
|-- public
| |-- .htaccess
| |-- favicon.gif
| |-- favicon.ico
| |-- passenger_wsgi.py
| `-- quickstart.html
`-- tmp
`-- restart.txt
And this is my .htaccess:
PassengerPython /home/dh_user/.pyenv/versions/3.5.2/envs/flask/bin/python
<VirtualHost *:80>
# This Flask web app will use Python 3.0
ServerName api.example.com
DocumentRoot /home/dh_user/api.example.com/public
</VirtualHost>
I have used all the guides below for help and reference:
https://help.dreamhost.com/hc/en-us/articles/215769548-Passenger-and-Python-WSGI
https://www.phusionpassenger.com/library/walkthroughs/start/python.html
https://www.byteconsole.com/get-flask-up-and-running-on-a-dreamhost-server-with-passenger/
Edit: Dreamhost doesn't allow me to look at the Passenger logs on a shared host. The best I can give you is my error.log, which, on every request, gives me
[Wed Apr 26 13:52:51 2017] [error] [client xxx.xxx.xxx.xxx] Premature end of script headers:
[Wed Apr 26 13:52:51 2017] [error] [client xxx.xxx.xxx.xxx] Premature end of script headers: internal_error.html
Edit 2: I'm using the Paste Python module to collect errors from Flask and it's giving me this:
File '/home/dh_user/.pyenv/versions/3.5.2/envs/flask/lib/python3.5/site-packages/paste/exceptions/errormiddleware.py', line 224 in next
return self.app_iterator.next()
AttributeError: 'ClosingIterator' object has no attribute 'next'
I added this to the top of my passenger_wsgi.py file and now the python errors go to my home directory:
import os,sys,os.path
errfile = open( '/home/user/error.log','a')
os.close(sys.stderr.fileno())
os.dup2(errfile.fileno(), sys.stderr.fileno())
Change /home/user/ to your dreamhost home directory, of course.
Well, I seem to have fixed it. What I can recall doing is removing Paste and the doing a few things after that.