My question looks a lot like this one but the accepted answer does not correspond to my target usage :
I would like to set url once and for all in an initialize.feature file, and never set it again afterwards. In other words I don't want to clutter every single feature files with the same following statement :
* url baseUrl
My baseUrl value is set based on karate.env, e.g. https://localhost for local environment and http://prod.env.com for prod. It does not change.
path will change in our feature files because we test different endpoints.
I tried the following setup :
in karate-config.js :
config.baseUrl = 'https://localhost';
// ... code changing config.baseUrl based on karate.env == 'prod' or not
var result = karate.callSingle('classpath:utility/initialize.feature', config);
in initialize.feature :
#ignore
Feature:
Scenario: Initialize
* print baseUrl
* url baseUrl
We can see that baseUrl is correctly printed when executing initialize.feature file.
But in any executed feature afterwards, I get the following error :
some-test.feature:24 - url not set, please refer to the keyword documentation for 'url'
Is it possible to set url only in my initialize.feature file, and never afterwards ?
Thanks.
No, you can't. You will have to do * url baseUrl at least once in every feature file. There are multiple reasons for this - readability and maintainability for one, and if you look at the "hello world example" - note how you could omit the url in the second call because you are following the REST-ful patterns.
Since you can do * url baseUrl in the Background: and have all other Scenario-s inherit - this is normally ok in practice, and in real-life API testing we see that you do need to switch URL-s within a test (e.g. for auth). If you feel very strongly about this - you could consider a pull-request. FWIW this is the first time in 2.5 years that someone has ever requested this.
Related
I am currently working on external app using Google Sheets and JSON for data transmission via Fetch API. I decided to mock the scenario (for debugging matters) then simple JSON comes from my external app through prepared Code.gs to be posted on Google sheets. The code snippet I run through Apps-scripts looks like this:
function _doPost(/* e */) {
// const body = e.postData.contents;
const bodyJSON = JSON.parse("{\"coords\" : \"123,456,789,112,113,114,115,116\"}" /* instead of : body */);
const db = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
db.getRange("A1:A10").setValue(bodyJSON.coords).setNumberFormat("#"); // get range, set value, set text format
}
The problem is the result I get: 123,456,789,112,113,000,000,000 As you see, starting from 114 and the later it outputs me 000,... instead. I thought, okay I am gonna explicitly specify format to be returned (saved) as a text format. If the output within the range selected on Google Sheets UI : Format -> Number -> it shows me Text.
However, interesting magic happens, let's say if I would update the body of the JSON to be parsed something like that when the sequence of numbers composed of 2 digits instead of 3 (notice: those are actual part of string, not true numbers, separated by comma!) : "{\"coords\" : \"123,456,789,112,113,114,115,116,17,18\"}" it would not only show response result as expected but also brings back id est fixes the "corrupted" values hidden under the 000,... as so : "{"coords" : "123,456,789,112,113,114,115,116,17,18 "}".
Even Logger.log() returns me initial JSON input as expected. I really have no clue what is going on. I would really appreciate one's correspondence to help solving this issue. Thank you.
You can try directly assigning a JSON formatted string in your bodyJSON variable instead of parsing a set of string using JSON.parse.
Part of your code should look like this:
const bodyJSON = {
"coords" : "123,456,789,112,113,114,115,116"
}
I found simple workaround after all: just added the preceding pair of zeros 0,0,123,... at the very beginning of coords. This prevents so called culprit I defined in my issue. If anyone interested, the external app I am building currently, it's called Hotspot widget : play around with DOM, append a marker which coordinates (coords) being pushed through Apps-script and saved to Google Sheets. I am providing a link with instructions on how to set up one's own copy of the app. It's a decent start-off for learning Vanilla JavaScript basics including simple database approach on the fly. Thank you and good luck!
Hotspot widget on Github
I'm working on some universal solution for problem with static files and updates in it.
Example: let's say there was site with /static/styles.css file - and site was used for a long time - so a lot of visitors cached this file in browser
Now we doing changes in this css file, and update on server, but some users still have old version (despite modification date returned by server)
The obvious solution is to add some version to file /static/styles.css?v=1.1 but in this case developer must track changes in this file and manually increase version
A second solution is to count the md5 hash of the file and add it to the url /static/styels.css/?v={mdp5hashvalue} which looks much better, but md5 should be calculated automatically somehow.
they possible way I see it - create some template tag like this
{% static_file "style.css" %}
which will render
<link src="/static/style.css?v=md5hash">
BUT, I do not want this tag to calculate md5 on every page load, and I do not want to store hash in django-cache, because then we will have to clear after updating file...
any thoughts ?
Django 1.4 now includes CachedStaticFilesStorage which does exactly what you need (well... almost).
Since Django 2.2 ManifestStaticFilesStorage should be used instead of CachedStaticFilesStorage.
You use it with the manage.py collectstatic task. All static files are collected from your applications, as usual, but this storage manager also creates a copy of each file with the MD5 hash appended to the name. So for example, say you have a css/styles.css file, it will also create something like css/styles.55e7cbb9ba48.css.
Of course, as you mentioned, the problem is that you don't want your views and templates calculating the MD5 hash all the time to find out the appropriate URLs to generate. The solution is caching. Ok, you asked for a solution without caching, I'm sorry, that's why I said almost. But there's no reason to reject caching, really. CachedStaticFilesStorage uses a specific cache named staticfiles. By default, it will use your existing cache system, and voilà! But if you don't want it to use your regular cache, perhaps because it's a distributed memcache and you want to avoid the overhead of network queries just to get static file names, then you can setup a specific RAM cache just for staticfiles. It's easier than it sounds: check out this excellent blog post. Here's what it would look like:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
'LOCATION': '127.0.0.1:11211',
},
'staticfiles': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'staticfiles-filehashes'
}
}
I would suggest using something like django-compressor. In addition to automatically handling this type of stuff for you, it will also automatically combine and minify your files for fast page load.
Even if you don't end up using it in entirety, you can inspect their code for guidance in setting up something similar. It's been better vetted than anything you'll ever get from a simple StackOverflow answer.
I use my own templatetag which add file modification date to url: https://bitbucket.org/ad3w/django-sstatic
Is reinventing the wheel and creating own implementation that bad? Furthermore I would like low level code (nginx for example) to serve my staticfiles in production instead of python application, even with backend. And one more thing: I'd like links stay the same after recalculation, so browser fetches only new files. So here's mine point of view:
template.html:
{% load md5url %}
<script src="{% md5url "example.js" %}"/>
out html:
static/example.js?v=5e52bfd3
settings.py:
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(PROJECT_DIR, 'static')
appname/templatetags/md5url.py:
import hashlib
import threading
from os import path
from django import template
from django.conf import settings
register = template.Library()
class UrlCache(object):
_md5_sum = {}
_lock = threading.Lock()
#classmethod
def get_md5(cls, file):
try:
return cls._md5_sum[file]
except KeyError:
with cls._lock:
try:
md5 = cls.calc_md5(path.join(settings.STATIC_ROOT, file))[:8]
value = '%s%s?v=%s' % (settings.STATIC_URL, file, md5)
except IsADirectoryError:
value = settings.STATIC_URL + file
cls._md5_sum[file] = value
return value
#classmethod
def calc_md5(cls, file_path):
with open(file_path, 'rb') as fh:
m = hashlib.md5()
while True:
data = fh.read(8192)
if not data:
break
m.update(data)
return m.hexdigest()
#register.simple_tag
def md5url(model_object):
return UrlCache.get_md5(model_object)
Note, to apply changes an uwsgi application (to be specific a process) should be restarted.
Django 1.7 added ManifestStaticFilesStorage, a better alternative to CachedStaticFilesStorage that doesn't use the cache system and solves the problem of the hash being computed at runtime.
Here is an excerpt from the documentation:
CachedStaticFilesStorage isn’t recommended – in almost all cases ManifestStaticFilesStorage is a better choice. There are several performance penalties when using CachedStaticFilesStorage since a cache miss requires hashing files at runtime. Remote file storage require several round-trips to hash a file on a cache miss, as several file accesses are required to ensure that the file hash is correct in the case of nested file paths.
To use it, simply add the following line to settings.py:
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
And then, run python manage.py collectstatic; it will append the MD5 to the name of each static file.
How about you always have a URL Parameter in your URL with a version and whenever you have a major release you change the version in your URL Parameter. Even in the DNS. So if www.yourwebsite.com loads up www.yourwebsite.com/index.html?version=1.0 then after the major release the browser should load www.yourwebsite.com/index.html?version=2.0
I guess this is similar to your solution 1. Instead of tracking files can you track whole directories? For example ratehr than /static/style/css?v=2.0 can you do /static-2/style/css or to make it even granular /static/style/cssv2/.
There is an update for #deathangel908 code. Now it works well with S3 storage also (and with any other storage I think). The difference is using of static file storage for getting file content. Original doesn't work on S3.
appname/templatetags/md5url.py:
import hashlib
import threading
from django import template
from django.conf import settings
from django.contrib.staticfiles.storage import staticfiles_storage
register = template.Library()
class UrlCache(object):
_md5_sum = {}
_lock = threading.Lock()
#classmethod
def get_md5(cls, file):
try:
return cls._md5_sum[file]
except KeyError:
with cls._lock:
try:
md5 = cls.calc_md5(file)[:8]
value = '%s%s?v=%s' % (settings.STATIC_URL, file, md5)
except OSError:
value = settings.STATIC_URL + file
cls._md5_sum[file] = value
return value
#classmethod
def calc_md5(cls, file_path):
with staticfiles_storage.open(file_path, 'rb') as fh:
m = hashlib.md5()
while True:
data = fh.read(8192)
if not data:
break
m.update(data)
return m.hexdigest()
#register.simple_tag
def md5url(model_object):
return UrlCache.get_md5(model_object)
The major advantage of this solution: you dont have to modify anything in the templates.
This will add the build version into the STATIC_URL, and then the webserver will remove it with a Rewrite rule.
settings.py
# build version, it's increased with each build
VERSION_STAMP = __versionstr__.replace(".", "")
# rewrite static url to contain the number
STATIC_URL = '%sversion%s/' % (STATIC_URL, VERSION_STAMP)
So the final url would be for example this:
/static/version010/style.css
And then Nginx has a rule to rewrite it back to /static/style.css
location /static {
alias /var/www/website/static/;
rewrite ^(.*)/version([\.0-9]+)/(.*)$ $1/$3;
}
Simple templatetag vstatic that creates versioned static files urls that extends Django's behaviour:
from django.conf import settings
from django.contrib.staticfiles.templatetags.staticfiles import static
#register.simple_tag
def vstatic(path):
url = static(path)
static_version = getattr(settings, 'STATIC_VERSION', '')
if static_version:
url += '?v=' + static_version
return url
If you want to automatically set STATIC_VERSION to the current git commit hash, you can use the following snippet (Python3 code adjust if necessary):
import subprocess
def get_current_commit_hash():
try:
return subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).strip().decode('utf-8')
except:
return ''
At settings.py call get_current_commit_hash(), so this will be calculated only once:
STATIC_VERSION = get_current_commit_hash()
I use a global base context in all my views, where I set the static version to be the millisecond time (that way, it will be a new version every time I restart my application):
# global base context
base_context = {
"title": settings.SITE_TITLE,
"static_version": int(round(time.time() * 1000)),
}
# function to merge context with base context
def context(items: Dict) -> Dict:
return {**base_context, **items}
# view
def view(request):
cxt = context({<...>})
return render(request, "page.html", cxt)
my page.html extends my base.html template, where I use it like this:
<link rel="stylesheet" type="text/css" href="{% static 'style.css' %}?v={{ static_version }}">
fairly simple and does the job
Similar question was asked before, but asked in an ambigous way and using a different code.
My problem: I want to make an exact copy of a .fits-file header into a new file. (I need to process a fits file in way, that I change the data, keep the header the same and save the result in a new file). Here a short example code, just demonstrating the tools I use and the discrepancy I arrive at:
data_old, header_old = fits.getdata("input_file.fits", header=True)
fits.writeto('output_file.fits', data_old, header_old, overwrite=True)
I would expect now that the the files are exact copies (headers and data of both being same). But if I check for difference, e.g. in this way -
fits.printdiff("input_file.fits", "output_file.fits")
I see that the two files are not exact copies of each other. The report says:
...
Files contain different numbers of HDUs:
a: 3
b: 2
Primary HDU:
Headers contain differences:
Headers have different number of cards:
a: 54
b: 4
...
Extension HDU 1:
Headers contain differences:
Keyword GCOUNT has different comments:
...
Why is there no exact copy? How can I do an exact copy of a header (and/or the data)? Is a key forgotten? Is there an alternative simple way of copy-pasting a fits-file-header?
If you just want to update the data array in an existing file while preserving the rest of the structure, have you tried the update function?
The only issue with that is it doesn't appear to have an option to write to a new file rather than update the existing file (maybe it should have this option). However, you can still use it by first copying the existing file, and then updating the copy.
Alternatively, you can do things more directly using the object-oriented API. Something like:
with fits.open(filename) as hdu_list:
hdu = hdu_list[<name or index of the HDU to update>]
hdu.data = <new ndarray>
# or hdu.data[<some index>] = <some value> i.e. just directly modify the existing array
hdu.writeto('updated.fits') # to write just that HDU to a new file, or
# hdu_list.writeto('updated.fits') # to write all HDUs, including the updated one, to a new file
There's nothing not "pythonic" about this :)
I have built a custom contenttype with an image field in Bolt 2.0.
image:
type: image
If no folder is specified the uploaded file goes to a folder named by the year-month
Result: 2014-11/myFileName.jpg
With the tag upload I can change this to something else.
image:
type: image
upload: "News/"
Result: News/myFileName.jpg
Is it possible to get the year-month folders after my costom path?
Result: News/2014-11/myFileName.jpg
The answer to this is yes, but not very simply so if you want a configurable way to do this you need to wait for 2.1 of Bolt where we're going to add variables to the upload: setting.
If you don't mind setting up your own bootstrap file and modifying the application then you can do it now.
The date prefix is generated by the $app['upload.prefix'] setting and currently returns the date string. What you need to do to modify this is change this to your own closure. I haven't tested this on a project so tweak if needed but after:
$app->initialize();
// Redefine the closure
$app['upload.prefix'] = function() {
$setting = $app['request']->get('handler');
$parts = explode('://', $setting);
$prefix = rtrim($parts[0], '/') . '/';
return $prefix.date('Y-m') . '/';
};
$app->run();
What we're doing here is reading the setting which is passed in the request and then concatenating the default date prefix onto the end of it.
As mentioned earlier 2.1 will see variable support introduced into the paths so options like
upload: news/{%month%}/{%day%}
upload: uploads/{%contenttype%}/{%id%}
will be easily definable in the contenttypes.yml file so If you don't mind waiting for a couple of months then this is obviously much simpler.
As of 3.2.9 this {%id%} principle doesn't seem to work yet ... :(
I'm trying to grab the href value in <a> HTML tags using Nokogiri.
I want to identify whether they are a path, file, URL, or even a <div> id.
My current work is:
hrefvalue = []
html.css('a').each do |atag|
hrefvalue << atag['href']
end
The possible values in a href might be:
somefile.html
http://www.someurl.com/somepath/somepath
/some/path/here
#previous
Is there a mechanism to identify whether the value is a valid full URL, or file, or path or others?
try URI:
require 'uri'
URI.parse('somefile.html').path
=> "somefile.html"
URI.parse('http://www.someurl.com/somepath/somepath').path
=> "/somepath/somepath"
URI.parse('/some/path/here').path
=> "/some/path/here"
URI.parse('#previous').path
=> ""
Nokogiri is often used with ruby's URI or open-uri, so if that's the case in your situation you'll have access to its methods. You can use that to attempt to parse the URI (using URI.parse). You can also generally use URI.join(base_uri, retrieved_href) to construct the full url, provided you've stored the base_uri.
(Edit/side-note: further details on using URI.join are available here: https://stackoverflow.com/a/4864170/624590 ; do note that URI.join that takes strings as parameters, not URI objects, so coerce where necessary)
Basically, to answer your question
Is there a mechanism to identify whether the value is a valid full
url, or file, or path or others?
If the retrieved_href and the base_uri are well formed, and retrieved_href == the joined pair, then it's an absolute path. Otherwise it's relative (again, assuming well formed inputs).
If you use URI to parse the href values, then apply some heuristics to the results, you can figure out what you want to know. This is basically what a browser has to do when it's about to send a request for a page or a resource.
Using your sample strings:
%w[
somefile.html
http://www.someurl.com/somepath/somepath
/some/path/here
#previous
].each do |u|
puts URI.parse(u).class
end
Results in:
URI::Generic
URI::HTTP
URI::Generic
URI::Generic
The only one that URI recognizes as a true HTTP URI is "http://www.someurl.com/somepath/somepath". All the others are missing the scheme "http://". (There are many more schemes you could encounter. See the specification for more information.)
Of the generic URIs, you can use some rules to sort through them so you'd know how to react if you have to open them.
If you gathered the HREF strings by scraping a page, you can assume it's safe to use the same scheme and host if the URI in question doesn't supply one. So, if you initially loaded "http://www.someurl.com/index.html", you could use "http://www.someurl.com/" as your basis for further requests.
From there, look inside the strings to determine whether they are anchors, absolute or relative paths. If the string:
Starts with # it's an anchor and would be applied to the current page without any need to reload it.
Doesn't contain a path delimiter /, it's a filename and would be added to the currently retrieved URL, substituting the file name, and retrieved. A nice way to do the substitution is to use File.dirname , File.basename and File.join against the string.
Begins with a path delimiter it's an absolute path and is used to replace the path in the original URL. URI::split and URI::join are your friends here.
Doesn't begin with a path delimiter, it's a relative path and is added to the current URI similarly to #2.
Regarding:
hrefvalue = []
html.css('a').each do |atag|
hrefvalue << atag['href']
end
I'd use this instead:
hrefvalue = html.search('a').map { |a| a['href'] }
But that's just me.
A final note: URI has some problems with age and needs an update. It's a useful library but, for heavy-duty URI rippin' apart, I highly recommend looking into using Addressable/URI.