I have some code that I have been porting from 2.7 to 3.6/3.7. Most of the unit tests, which have a pretty good coverage, already execute successfully under 3.x. But I have yet to fully commit to switching over to 3.x for development.
I recently noticed, when running black - the code formatter that it chokes if my code would not compile under 3.x, with a message about 3.6 AST-based parsing failing.
Is black a reliable indicator of 3.x-readiness, at least at the syntax level? I know that 2to3 is the tool to use. And I know that for example, it would not catch differences in the standard library (basestring disappearing, StringIO.StringIO becoming io.StringIO, etc...).
but it seems nice that a code formatter could incidentally help out as well.
very basic sample, invalid syntax for 3.x:
print "a", 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21
gives:
error: cannot format test_black.py:
cannot use --safe with this file; failed to parse source file with
Python 3.6's builtin AST.
Re-run with --fast or stop using deprecated Python 2 syntax.
AST error message: Missing parentheses in call to 'print'.
Did you mean print("a", 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21)? (<unknown>, line 1)
All done! 💥 💔 💥
1 file failed to reformat.
fix the syntax to 3.x and it works.
If I do the right thing, and add parenthesis print ("a", 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21), then all's well:
reformatted test_black.py
All done! ✨ 🍰 ✨
1 file reformatted.
Related
Develop a Python method change(amount) that for any integer amount in the range from 24 to 1000 returns a list consisting of numbers 5 and 7 only, such that their sum is equal to amount. For example, change(28) may return [7, 7, 7, 7], while change(49) may return [7, 7, 7, 7, 7, 7, 7] or [5, 5, 5, 5, 5, 5, 5, 7, 7] or [7, 5, 5, 5, 5, 5, 5, 5, 7].
To solve this quiz, implement the method change(amount) on your machine, test it on several inputs, and then paste your code in the field below and press the submit quiz button. Your submission should contain the change method only (in particular, make sure to remove all print statements).
Just started programming, quite proud of this. Here you go:
To use: print(change(amount))
def change(amount):
if amount < 24 or amount > 1000:
return 'error'
array = []
while True:
if (amount/5).is_integer():
for i in range(int(amount/5)):
array.append(5)
return array
array.append(7)
amount += -7
while amount > 0:
break
I have an old vb6 program which queries an access 2000 database. I have a fairly long query which looks something like this:
Select * from table where key in ( 0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 13, 14, 15, 19, 20, 21, 24, 27, 29, 30, 35, 38, 39, 40, 42, 43, 44, 46, 47, 49, 50, 53, 56, 59, 60, 61, 63, 64, 65, 66, 67, 68, 72, 76, 80, 84, 86, 89, 90, 91, 93, 94, 98, 99, 10041, 10042, 10045, 10046, 10047, 10049, 10057, 10060, 10089, 32200, 32202, 32203, 32204, 32205, 32207, 32214, 32245, 32303, 32314, 32403, 32405, 32414, 32415, 32503, 32703, 32803, 32903, 33003, 33014, 33102, 33103, 33303, 33403, 33405, 33601, 33603, 33604, 33614, 33705, 33714, 33901, 33903, 33914, 34001, 34105, 34114, 34203, 34303, 34401, 34501, 34601, 34603, 34604, 34605, 34803, 41001, 41005, 41007, 41013, 42001, 42005, 42007, 42013, 43001, 43002, 44001, 44007, 46001, 46007, 99999, 9999999)
However, when I look at the RecordSource of the data object, it seems that the query is being truncated to this (which is obviously not syntactically valid and throws an error):
Select * from table where key in ( 0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 13, 14, 15, 19, 20, 21, 24, 27, 29, 30, 35, 38, 39, 40, 42, 43, 44, 46, 47, 49, 50, 53, 56, 59, 60, 61, 63, 64, 65, 66, 67, 68, 72, 76, 80, 84, 86, 89, 90, 91, 93, 94, 98, 99, 100
My data source looks like this:
Begin VB.Data dtaList
Caption = "dtaList"
Connect = "Access 2000;"
DatabaseName = ""
DefaultCursorType= 0 'DefaultCursor
DefaultType = 2 'UseODBC
Exclusive = 0 'False
Height = 345
Left = 960
Options = 0
ReadOnly = 0 'False
RecordsetType = 1 'Dynaset
RecordSource = ""
Top = 4440
Visible = 0 'False
Width = 2295
End
I've tried running the full query in the access database itself which works fine.
Is this a limitation in the VB.Data object, or is there some other explanation? Is there any way I can get around this issue?
Unfortunately I am unable to upgrade to a newer version of access.
The truncated version of the SQL statement you posted is 246 characters long, so it appears that something along the line is limiting the length of the SQL string to somewhere around 255 characters. As you have discovered by pasting the query into Access itself, the actual size limit of an Access query string is much larger (around 64,000 characters, I believe).
I remember running across a similar issue years ago but my problem was an INSERT statement that was writing some rather long strings to the database. The workaround in that case was to use a parameter query (which I realize, in hindsight, that I should have been using anyway). It greatly shortened the length of the SQL string because the parameters were passed separately. Unfortunately that workaround probably wouldn't help you because even if you dynamically created a parameterized version of the query it wouldn't be all that much shorter than the current SQL string.
Another workaround would be to write all of those numbers for the IN clause as rows in a temporary table named something like [inValues], and then use the query
SELECT [table].*
FROM
[table]
INNER JOIN
[inValues]
ON [table].[key] = [inValues].[key]
i have a dataset of about 1.85 GB which contains h5 files,i need to process these files using hadoop,for this i may need to convert these files to text or csv.
is there any way hadoop can read h5 files?or any good online tool to convert h5 files to csv or text files?or can any one give a link where i can download a huge dataset which contains text or csv files?
thanks in advance
Have you tried OPeNDAP Hyrax server with hdf5_handler module?
For example, from the sample HDF5 file [1], you can get the following ASCII data [2]:
Dataset: grid_1_2d.h5
temperature[0], 10, 10, 10, 10, 10, 10, 10, 10
temperature[1], 11, 11, 11, 11, 11, 11, 11, 11
temperature[2], 12, 12, 12, 12, 12, 12, 12, 12
temperature[3], 13, 13, 13, 13, 13, 13, 13, 13
...
OPeNDAP Hyrax server with hdf5_handler is a great tool/service because you can select (and subset) a dataset from an HDF5 file easily using HTML form as well [3]. You can find the detailed information about OPeNDAP hdf5_handler from [4].
[1] http://eosdap.hdfgroup.org:8080/opendap/data/hdf5/grid_1_2d.h5
[2] http://eosdap.hdfgroup.org:8080/opendap/data/hdf5/grid_1_2d.h5.ascii
[3] http://eosdap.hdfgroup.org:8080/opendap/data/hdf5/grid_1_2d.h5.html
[4] http://hdfeos.org/software/hdf5_handler.php
I have some data files with content
a1 b1 c1 d1
a1 b2 c2 d2
...
[blank line]
a2 b1 c1 d1
a2 b2 c2 d2
...
I plot this with gnuplot using
splot 'file' u 1:2:3:4 w pm3d.
Now, I want to use a binary file. I created the file with Fortran using unformatted stream-access (direct or sequential access did not work directly). By using gnuplot with
splot 'file' binary format='%float%float%float%float' u 1:2:3
I get a normal 3D-plot. However, the pm3d-command does not work as I don't have the blank lines in the binary file. I get the error message:
>splot 'file' binary format='%float%float%float%float' u 1:2:3:4 w pm3d
Warning: Single isoline (scan) is not enough for a pm3d plot.
Hint: Missing blank lines in the data file? See 'help pm3d' and FAQ.
According to the demo script in http://gnuplot.sourceforge.net/demo/image2.html, I have to specify the record length (which I still don't understand right). However, using this script from the demo page and the command with pm3d obtains the same error message:
splot 'scatter2.bin' binary record=30:30:29:26 u 1:2:3 w pm3d
So how is it possible to plot this four dimensional data from a binary file correctly?
Edit: Thanks, mgilson. Now it works fine. Just for the record: My fortran code-snippet:
open(unit=83,file=fname,action='write',status='replace',access='stream',form='unformatted')
a= 0.d0
b= 0.d0
do i=1,200
do j=1,100
write(83)real(a),real(b),c(i,j),d(i,j)
b = b + db
end do
a = a + da
b = 0.d0
end do
close(83)
The gnuplot commands:
set pm3d map
set contour
set cntrparam levels 20
set cntrparam bspline
unset clabel
splot 'fname' binary record=(100,-1) format='%float' u 1:2:3:4 t 'd as pm3d-projection, c as contour'
Great question, and thanks for posting it. This is a corner of gnuplot I hadn't spent much time with before. First, I need to generate a little test data -- I used python, but you could use fortran just as easily:
Note that my input array (b) is just a 10x10 array. The first two "columns" in the datafile are just the index (i,j), but you could use anything.
>>> import numpy as np
>>> a = np.arange(10)
>>> b = a[None,:]+a[:,None]
>>> b
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
[ 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
[ 3, 4, 5, 6, 7, 8, 9, 10, 11, 12],
[ 4, 5, 6, 7, 8, 9, 10, 11, 12, 13],
[ 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
[ 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
[ 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
[ 8, 9, 10, 11, 12, 13, 14, 15, 16, 17],
[ 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]])
>>> with open('foo.dat','wb') as foo:
... for (i,j),dat in np.ndenumerate(b):
... s = struct.pack('4f',i,j,dat,dat)
... foo.write(s)
...
So here I just write 4-floating point values to the file for each data-point. Again, this is what you've already done using fortran. Now for plotting it:
splot 'foo.dat' binary record=(10,-1) format='%float' u 1:2:3:4 w pm3d
I believe that this specifies that each "scan" is a "record". Since I know that each scan will be 10 floats long, that becomes the first index in the record list. The -1 indicates that gnuplot should keep reading records until it finds the end of the file.
I am wondering if Cuda would be useful for this type of problem(and how to approach it in Cuda). Basically I have been using python to find combinations of a list but as the data gets large I'm thinking running it on a gpu maybe an interesting idea.
Say I have a list [1, 2, 3,4,5,6,7,8] and I want only 7 combinations then I would get:
(1, 2, 3, 4, 5, 6, 7)
(1, 2, 3, 4, 5, 6, 8)
(1, 2, 3, 4, 5, 7, 8)
(1, 2, 3, 4, 6, 7, 8)
(1, 2, 3, 5, 6, 7, 8)
(1, 2, 4, 5, 6, 7, 8)
(1, 3, 4, 5, 6, 7, 8)
(2, 3, 4, 5, 6, 7, 8)
As the data gets larger it takes a long time. I have been using itertools.combinations which abstracts everything from me so if I try to program this myself is there any resources or proxy code I can look at? Most of the algorithms related to combinations are recursive and my Cuda card does not support recursions.
Any suggestion/tips on where to start?
I have done a small CUDA project that does bin packing by trying permutations:
http://www.dahlsys.com/software/fill_media/index.html