I started with reverse engineering and using the IDA disassembler tool.
I wrote some programs in C++, made an .exe and reversed it in IDA to "hack" my own programs.
Now I wanted to do the same with a python program. As a start a made this simple program:
inp = input("What is your name?\n")
print("Hi", inp)
Then I built an .exe using:
pyinstaller main.py
Loaded it into IDA and searched for strings. IDA gives me a lot of strings but "What is your name" is not in its list. Why is that?
Yes, pyinstaller builds an executable but it is not a "normal" executable. Your Python code is actually in a compressed archive.
$ readelf -S main
There are 30 section headers, starting at offset 0x1ad400:
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[ 0] NULL 0000000000000000 00000000
0000000000000000 0000000000000000 0 0 0
[ 1] .interp PROGBITS 00000000000002a8 000002a8
000000000000001c 0000000000000000 A 0 0 1
[ 2] .note.ABI-tag NOTE 00000000000002c4 000002c4
0000000000000020 0000000000000000 A 0 0 4
[ 3] .note.gnu.bu[...] NOTE 00000000000002e4 000002e4
0000000000000024 0000000000000000 A 0 0 4
[ 4] .gnu.hash GNU_HASH 0000000000000308 00000308
0000000000000034 0000000000000000 A 5 0 8
[ 5] .dynsym DYNSYM 0000000000000340 00000340
00000000000007b0 0000000000000018 A 6 1 8
[ 6] .dynstr STRTAB 0000000000000af0 00000af0
0000000000000348 0000000000000000 A 0 0 1
[ 7] .gnu.version VERSYM 0000000000000e38 00000e38
00000000000000a4 0000000000000002 A 5 0 2
[ 8] .gnu.version_r VERNEED 0000000000000ee0 00000ee0
00000000000000a0 0000000000000000 A 6 3 8
[ 9] .rela.dyn RELA 0000000000000f80 00000f80
0000000000000198 0000000000000018 A 5 0 8
[10] .rela.plt RELA 0000000000001118 00001118
00000000000006d8 0000000000000018 AI 5 24 8
[11] .init PROGBITS 0000000000002000 00002000
0000000000000017 0000000000000000 AX 0 0 4
[12] .plt PROGBITS 0000000000002020 00002020
00000000000004a0 0000000000000010 AX 0 0 16
[13] .plt.got PROGBITS 00000000000024c0 000024c0
0000000000000008 0000000000000008 AX 0 0 8
[14] .text PROGBITS 00000000000024d0 000024d0
0000000000003f21 0000000000000000 AX 0 0 16
[15] .fini PROGBITS 00000000000063f4 000063f4
0000000000000009 0000000000000000 AX 0 0 4
[16] .rodata PROGBITS 0000000000007000 00007000
0000000000001618 0000000000000000 A 0 0 8
[17] .eh_frame_hdr PROGBITS 0000000000008618 00008618
000000000000025c 0000000000000000 A 0 0 4
[18] .eh_frame PROGBITS 0000000000008878 00008878
0000000000000e90 0000000000000000 A 0 0 8
[19] .init_array INIT_ARRAY 000000000000ad70 00009d70
0000000000000008 0000000000000008 WA 0 0 8
[20] .fini_array FINI_ARRAY 000000000000ad78 00009d78
0000000000000008 0000000000000008 WA 0 0 8
[21] .data.rel.ro PROGBITS 000000000000ad80 00009d80
0000000000000040 0000000000000000 WA 0 0 32
[22] .dynamic DYNAMIC 000000000000adc0 00009dc0
0000000000000210 0000000000000010 WA 6 0 8
[23] .got PROGBITS 000000000000afd0 00009fd0
0000000000000028 0000000000000008 WA 0 0 8
[24] .got.plt PROGBITS 000000000000b000 0000a000
0000000000000260 0000000000000008 WA 0 0 8
[25] .data PROGBITS 000000000000b260 0000a260
0000000000000010 0000000000000000 WA 0 0 8
[26] .bss NOBITS 000000000000b280 0000a270
00000000000172c0 0000000000000000 WA 0 0 32
[27] .comment PROGBITS 0000000000000000 0000a270
000000000000001c 0000000000000001 MS 0 0 1
[28] pydata PROGBITS 0000000000000000 0000a28c
00000000001a3066 0000000000000000 0 0 1
[29] .shstrtab STRTAB 0000000000000000 001ad2f2
000000000000010b 0000000000000000 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
l (large), p (processor specific)
$
Note the pydata section!
You can use archive_viewer.py to view the archives in main:
$ python archive_viewer.py main
pos, length, uncompressed, iscompressed, type, name
[(0, 215, 285, 1, 'm', 'struct'),
(215, 1059, 1792, 1, 'm', 'pyimod01_os_path'),
(1274, 4080, 8938, 1, 'm', 'pyimod02_archive'),
(5354, 5518, 13005, 1, 'm', 'pyimod03_importers'),
(10872, 1825, 4051, 1, 's', 'pyiboot01_bootstrap'),
(12697, 1161, 2135, 1, 's', 'pyi_rth_multiprocessing'),
(13858, 125, 143, 1, 's', 'main'),
(13983, 1701919, 1701919, 0, 'z', 'PYZ-00.pyz')]
? x main
to filename? main.x
? q
$ xxd main.x
00000000: e300 0000 0000 0000 0000 0000 0000 0000 ................
00000010: 0003 0000 0040 0000 0073 1600 0000 6500 .....#...s....e.
00000020: 6400 8301 5a01 6502 6401 6501 8302 0100 d...Z.e.d.e.....
00000030: 6402 5300 2903 7a13 5768 6174 2069 7320 d.S.).z.What is
00000040: 796f 7572 206e 616d 653f 0ada 0248 694e your name?...HiN
00000050: 2903 da05 696e 7075 74da 0369 6e70 da05 )...input..inp..
00000060: 7072 696e 74a9 0072 0500 0000 7205 0000 print..r....r...
00000070: 007a 076d 6169 6e2e 7079 da08 3c6d 6f64 .z.main.py..<mod
00000080: 756c 653e 0100 0000 7302 0000 0008 01 ule>....s......
$
As you can see from the above output the string you are looking for is stored in a compressed archive called main which in turn is part of the main executable build by pyinstaller.
Related
Assume the following df:
ib c d1 d2
0 1.14 1 1 0
1 1.0 1 1 0
2 0.71 1 1 0
3 0.6 1 1 0
4 0.66 1 1 0
5 1.0 1 1 0
6 1.26 1 1 0
7 1.29 1 1 0
8 1.52 1 1 0
9 1.31 1 1 0
10 0.89 1 0 1
d1 and d2 are perfectly colinear. Now I estimate the following regression model:
import statsmodels.api as sm
reg = sm.OLS(df['ib'], df[['c', 'd1', 'd2']]).fit().summary()
reg
This gives me the following output:
<class 'statsmodels.iolib.summary.Summary'>
"""
OLS Regression Results
==============================================================================
Dep. Variable: ib R-squared: 0.087
Model: OLS Adj. R-squared: -0.028
Method: Least Squares F-statistic: 0.7590
Date: Thu, 17 Nov 2022 Prob (F-statistic): 0.409
Time: 12:19:34 Log-Likelihood: -1.5470
No. Observations: 10 AIC: 7.094
Df Residuals: 8 BIC: 7.699
Df Model: 1
Covariance Type: nonrobust
===============================================================================
coef std err t P>|t| [0.025 0.975]
-------------------------------------------------------------------------------
c 0.7767 0.111 7.000 0.000 0.521 1.033
d1 0.2433 0.127 1.923 0.091 -0.048 0.535
d2 0.5333 0.213 2.499 0.037 0.041 1.026
==============================================================================
Omnibus: 0.257 Durbin-Watson: 0.760
Prob(Omnibus): 0.879 Jarque-Bera (JB): 0.404
Skew: 0.043 Prob(JB): 0.817
Kurtosis: 2.019 Cond. No. 8.91e+15
==============================================================================
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The smallest eigenvalue is 2.34e-31. This might indicate that there are
strong multicollinearity problems or that the design matrix is singular.
"""
However, including c, d1 and d2 represents the well known dummy variable trap which, from my understanding, should make it impossible to estimate the model. Why is this not the case here?
I wrote a scenario in ns2 software that has 5 fixed nodes and 1 mobile sink.
The mobile sink should collect the nodes' information.
I want to calculate the mean avg (energy_level ,number of nodes & traffic's data on nodes) and store it in the mobile sink.
The my code that I wrote does not any error but it does not print out.
My code & trace file are in attachment
I have to calculate this avg with proc in tcl script code not with awk code.Please help me.
proc create_common_app {destination_id disseminating_type disseminating_interval energy_level } {
global val
#puts "executing create_common_app\n"
if { $val(commonApp) == "Application/SensorBaseApp/CommonNodeApp" }
{
set app_ [new $val(commonApp)]
$app_ set destination_id_ $destination_id
}
$app_ set disseminating_type_ $disseminating_type
$app_ set disseminating_interval_ $disseminating_interval
$app_ set energy_level _ $energy_level
return $app_
}
proc create_mobile_sink_app { source _id disseminating_type disseminating_interval energy_level outside_network} {
global val
set app_ [new $val(sinkApp)]
$app_ set source _id _ $source _id
$app_ set disseminating_type_ $disseminating_type
$app_ set disseminating_interval_ $disseminating_interval
$app_ set energy_level _ $energy_level
$app_ set outside_network_ $outside_network
#$app_ set avg_ $avg
return $app_
}
#########################################################################
#Me
####################################################################
proc avg { energy_level nn disseminating_interval } {
set avgen 0
for {set i 0} {$i < 5} {incr i} {
set $avgen [expr $energy_level(i) + $avgen ]
}
set $avgen [expr $avgen / $nn]
return $avgen
set xl [list $avgen $nn $disseminating_interval]
}
proc ArithmeticMean { $xl } {
set length [llength $xl]
if {$length == 0} {
return 0.0
}
set sum [::tcl::mathop::+ {*}$xl]
return [expr {double($sum) / $length}]
set avgt [expr {double($sum) / $length}]
puts "average is $avgt_"
}
#####################################################################
tracefile
M 0.00000 0 (9.00, 5.00, 0.00), (0.00, 0.00), 0.00
s 5.000000000 _1_ AGT --- 0 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [0] 0 0
r 5.000000000 _1_ RTR --- 0 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [0] 0 0
s 5.000000000 _1_ AGT --- 1 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [1] 0 0
r 5.000000000 _1_ RTR --- 1 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [1] 0 0
s 5.000000000 _1_ AGT --- 2 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [2] 0 0
r 5.000000000 _1_ RTR --- 2 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [2] 0 0
s 5.000000000 _1_ AGT --- 3 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [3] 0 0
r 5.000000000 _1_ RTR --- 3 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [3] 0 0
s 5.000000000 _1_ AGT --- 4 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [4] 0 0
r 5.000000000 _1_ RTR --- 4 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [4] 0 0
s 5.000000000 _1_ AGT --- 5 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [5] 0 0
r 5.000000000 _1_ RTR --- 5 cbr 150 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [5] 0 0
s 5.000000000 _1_ AGT --- 6 cbr 100 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [6] 0 0
r 5.000000000 _1_ RTR --- 6 cbr 100 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:0 2:0 32 0] [6] 0 0
s 5.000000000 _1_ RTR --- 0 AODV 48 [0 0 0 0] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:255 -1:255 30 0] [0x2 1 1 [2 0] [1 4]] (REQUEST)
s 5.000535000 _1_ MAC --- 0 AODV 106 [0 ffffffff 2 800] [energy 0.050000 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:255 -1:255 30 0] [0x2 1 1 [2 0] [1 4]] (REQUEST)
N -t 5.000535 -n 0 -e 0.049980
N -t 5.000535 -n 4 -e 0.049980
N -t 5.000535 -n 2 -e 0.049980
N -t 5.000535 -n 3 -e 0.049980
r 5.001383005 _0_ MAC --- 0 AODV 48 [0 ffffffff 2 800] [energy 0.049980 ei 0.000 es 0.000 et 0.000 er 0.000] ------- [1:255 -1:255 30 0] [0x2 1 1 [2 0] [1 4]] (REQUEST)
r 5.001383013 _4_ MAC --- 0 AODV 48 [0 ffffffff 2 800] [energy 0.049980 ei 0.000 es 0.000 et 0.000
.
.
.....
proc avg { energy_level nn disseminating_interval } {
set avgen 0
for {set i 0} {$i < 5} {incr i} {
set $avgen [expr $energy_level(i) + $avgen ]
}
set $avgen [expr $avgen / $nn]
return $avgen
set xl [list $avgen $nn $disseminating_interval]
}
Notes:
You can't pass arrays by value: you need to pass the array name and use upvar to connect the name to the actual array in the caller's context
Use braces around your expr expressions.
You need to dereference $i in the array expansion.
Don't use $ in the first argument to set
do you want to use $nn in the for loop, or the hard-coded value 5?
Corrected code
proc avg { energy_level_var nn disseminating_interval } {
upvar 1 $energy_level_var energy_level
if {$nn == 0} {
error "denominator cannot be zero"
}
set sum 0
for {set i 0} {$i < $nn} {incr i} {
set sum [expr {$energy_level($i) + $sum}]
}
return [expr {$sum / $nn}]
}
And call like this
array set energy_level {0 1.0 1 1.1 2 1.2 3 1.3 4 1.4 5 1.5}
puts "average is: [avg energy_level 6 "?"]" ;# average is: 1.25
Also, don't use $ in the declaration of the ArithmeticMean proc:
proc ArithmeticMean { xl } {
Client side operating System: Qnx 6.4.0
Server side: Red Hat 6.5
I trying to connect to mysql server at boot time from the following script (at qnx machine).
#!/bin/sh
cd /root
io-pkt-v4-hc -di82544 -ptcpip
ifconfig en0 192.168.4.122 netmask 255.255.0.0
slay qconn
qconn
dumper -n
inetd
devc-pty
if_up -p en0
./mysql_connect
Source code of mysql_connect
MYSQL *conn;
MYSQL_RES *res;
MYSQL_ROW row;
const char *database = "PRC_OPT";
unsigned int conn_timeout = 5;
conn = mysql_init(NULL);
Wprintf(Terminal2,"MySQL Tables connection = %p:\n", conn);
mysql_options(conn, MYSQL_OPT_CONNECT_TIMEOUT, &conn_timeout);
/* Connect to database */
if (!mysql_real_connect(conn, DATABASE_IPADDRESS,
USER, PASSWORD, database, DATABASE_PORT, NULL, 0)) {
Wprintf(Terminal2, "%s\n", mysql_error(conn));
mysql_close(conn);
return;
}
/* send SQL query */
if (mysql_query(conn, "SELECT * FROM ADRLIS WHERE PM='TKPRC' AND NUMTS=1;")) {
Wprintf(Terminal2, "%s\n", mysql_error(conn));
return;
}
res = mysql_use_result(conn);
/* close connection */
mysql_free_result(res);
mysql_close(conn);
As result I have a following error at mysql_real_connect
Can't connect to MySql server on 'ip address' (4)
but then .............when I invoke ./mysql_connect (at second time from bash command line) all works correctly.
At first time I was thinking that iptables rejected my connection packets, but this situation is repeated even so I do service iptables stop.
I have invoked tcpdump at server side and have following
14:06:15.402347 IP 192.168.4.122.65534 > prc.mysql: Flags [S], seq 33833343, win 32768, options [mss 1460,nop,wscale 0,sackOK,nop,nop,nop,nop,TS val 0 ecr 0], length 0
0x0000: 5254 00b2 fdb1 0023 6b01 13d8 0800 4500 RT.....#k.....E.
0x0010: 0040 ed1b 4000 4006 c309 c0a8 047a c0a8 .#..#.#......z..
0x0020: 04c8 fffe 0cea 0204 417f 0000 0000 b002 ........A.......
0x0030: 8000 db01 0000 0204 05b4 0103 0300 0402 ................
0x0040: 0101 0101 080a 0000 0000 0000 0000 ..............
14:06:15.403234 IP prc.mysql > 192.168.4.122.65534: Flags [S.], seq 4145356488, ack 33833344, win 14480, options [mss 1460,sackOK,TS val 703030136 ecr 0,nop,wscale 7], length 0
0x0000: 0023 6b01 13d8 5254 00b2 fdb1 0800 4500 .#k...RT......E.
0x0010: 003c 0000 4000 4006 b029 c0a8 04c8 c0a8 .<..#.#..)......
0x0020: 047a 0cea fffe f715 1ec8 0204 4180 a012 .z..........A...
0x0030: 3890 9122 0000 0204 05b4 0402 080a 29e7 8.."..........).
0x0040: 6378 0000 0000 0103 0307 cx........
14:06:15.403451 IP 192.168.4.122.65534 > prc.mysql: Flags [R], seq 33833344, win 0, length 0
0x0000: 5254 00b2 fdb1 0023 6b01 13d8 0800 4500 RT.....#k.....E.
0x0010: 0028 ed1c 0000 4006 0321 c0a8 047a c0a8 .(....#..!...z..
0x0020: 04c8 fffe 0cea 0204 4180 0000 0000 5004 ........A.....P.
0x0030: 0000 d4e0 0000 0000 0000 0000 ............
As we can see the server reset tcp connection from client. If we try to invoke second time from the bash command line tcp connection is established successfully and then we can work with database without any problems
14:18:09.068632 IP 192.168.4.122.65533 > 192.168.4.200.3306: Flags [S], seq 917705451, win 32768, options [mss 1460,nop,wscale 0,sackOK,nop,nop,nop,nop,TS val 0 ecr 0], length 0
0x0000: 5254 00b2 fdb1 0023 6b01 13d8 0800 4500 RT.....#k.....E.
0x0010: 0040 f115 4000 4006 bf0f c0a8 047a c0a8 .#..#.#......z..
0x0020: 04c8 fffd 0cea 36b3 12eb 0000 0000 b002 ......6.........
0x0030: 8000 d4e7 0000 0204 05b4 0103 0300 0402 ................
0x0040: 0101 0101 080a 0000 0000 0000 0000 ..............
14:18:09.068681 IP 192.168.4.200.3306 > 192.168.4.122.65533: Flags [S.], seq 4004872844, ack 917705452, win 14480, options [mss 1460,sackOK,TS val 703743802 ecr 0,nop,wscale 7], length 0
0x0000: 0023 6b01 13d8 5254 00b2 fdb1 0800 4500 .#k...RT......E.
0x0010: 003c 0000 4000 4006 b029 c0a8 04c8 c0a8 .<..#.#..)......
0x0020: 047a 0cea fffd eeb5 828c 36b3 12ec a012 .z........6.....
0x0030: 3890 4bd7 0000 0204 05b4 0402 080a 29f2 8.K...........).
0x0040: 473a 0000 0000 0103 0307 G:........
14:18:09.068920 IP 192.168.4.122.65533 > 192.168.4.200.3306: Flags [.], ack 1, win 33580, options [nop,nop,TS val 0 ecr 703743802], length 0
0x0000: 5254 00b2 fdb1 0023 6b01 13d8 0800 4500 RT.....#k.....E.
0x0010: 0034 f116 4000 4006 bf1a c0a8 047a c0a8 .4..#.#......z..
0x0020: 04c8 fffd 0cea 36b3 12ec eeb5 828d 8010 ......6.........
0x0030: 832c 3007 0000 0101 080a 0000 0000 29f2 .,0...........).
0x0040: 473a G:
What is the difference between those two starts??? Why the first connection is rejected???
I have a REST api with DRF and start to see already a performance hit with 100 objects and 1 user requesting (me - testing).
When requesting the more complex query, I get these results for CPU, always 5 - 10s:
Resource Value
>User CPU time 5987.089 msec
System CPU time 463.929 msec
Total CPU time 6451.018 msec
Elapsed time 6800.938 msec
Context switches 9 voluntary, 773 involuntary
but the SQL query stays below 100 ms
The more simple queries show similar behaviour, with CPU times around 1s and query time around 20 ms
So far, what I have tried out:
I am doing select_related() and prefetch_related(), which did improve the query time but not CPU time
I am using Imagekit to generate pictures, on a S3 instance. I removed the whole specification to test and this had minor impact
I run a method field to fetch user-specific data. Removing this had only minor impact
I have checked logs files on the backend and nothing specific shows up here...
Backend is Nginx - supervisord - gunicorn - postgresql - django 1.8.1
Here are the serializer and view:
class ParticipationOrganizationSerializer(ModelSerializer):
organization = OrganizationSerializer(required=False, read_only=True, )
bookmark = SerializerMethodField(
required=False,
read_only=True,
)
location_map = LocationMapSerializer(
required=False,
read_only=True,
)
class Meta:
model = Participation
fields = (
'id',
'slug',
'organization',
'location_map',
'map_code',
'partner',
'looking_for',
'complex_profile',
'bookmark',
'confirmed',
)
read_only_fields = (
'id',
'slug',
'organization',
'location_map',
'map_code',
'partner',
'bookmark',
'confirmed',
)
def get_bookmark(self, obj):
request = self.context.get('request', None)
if request is not None:
if(request.user.is_authenticated()):
# print(obj.bookmarks.filter(author=request.user).count())
try:
bookmark = obj.bookmarks.get(author=request.user)
# bookmark = Bookmark.objects.get(
# author=request.user,
# participation=obj,
# )
return BookmarkSerializer(bookmark).data
except Bookmark.DoesNotExist:
# We have nothing yet
return None
except Bookmark.MultipleObjectsReturned:
# This should not happen, but in case it does, delete all
# the bookmarks for safety reasons.
Bookmark.objects.filter(
author=request.user,
participation=obj,
).delete()
return None
return None
class ParticipationOrganizationViewSet(ReadOnlyModelViewSet):
"""
A readonly ViewSet for viewing participations of a certain event.
"""
serializer_class = ParticipationOrganizationSerializer
queryset = Participation.objects.all().select_related(
'location_map',
'organization',
'organization__logo_image',
).prefetch_related(
'bookmarks',
)
lookup_field = 'slug'
def get_queryset(self):
event_slug = self.kwargs['event_slug']
# Filter for the current event
# Filter to show only the confirmed participations
participations = Participation.objects.filter(
event__slug=event_slug,
confirmed=True
).select_related(
'location_map',
'organization',
'organization__logo_image',
).prefetch_related(
'bookmarks',
)
# Filter on partners? This is a parameter passed on in the url
partners = self.request.query_params.get('partners', None)
if(partners == "true"):
participations = participations.filter(partner=True)
return participations
# http://stackoverflow.com/questions/22616973/django-rest-framework-use-different-serializers-in-the-same-modelviewset
def get_serializer_class(self):
if self.action == 'list':
return ParticipationOrganizationListSerializer
if self.action == 'retrieve':
return ParticipationOrganizationSerializer
return ParticipationOrganizationListSerializer
Any help is very much appreciated!
update
I dumped the data to my local machine and I am observing similar times. I guess this rules out the whole production setup (nginx, gunicorn)?
update 2
Here are the results of the profiler.
Also I made some progress in improving the speeds by
Simplifying my serializers
Doing the tests with curl and having Debug Toolbar off
ncalls tottime percall cumtime percall filename:lineno(function)
0 0 0 profile:0(profiler)
1 0 0 3.441 3.441 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/views.py:442(dispatch)
1 0 0 3.441 3.441 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/viewsets.py:69(view)
1 0 0 3.441 3.441 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/django/views/decorators/csrf.py:57(wrapped_view)
1 0 0 3.44 3.44 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/mixins.py:39(list)
1 0 0 3.438 3.438 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:605(to_representation)
1 0 0 3.438 3.438 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:225(data)
1 0 0 3.438 3.438 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:672(data)
344/114 0.015 0 3.318 0.029 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:454(to_representation)
805 0.01 0 2.936 0.004 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/fields.py:1368(to_representation)
2767 0.013 0 2.567 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py:166(send)
2070 0.002 0 2.52 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/registry.py:52(existence_required_receiver)
2070 0.005 0 2.518 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/registry.py:55(_receive)
2070 0.004 0 2.513 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/utils.py:147(call_strategy_method)
2070 0.002 0 2.508 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/strategies.py:14(on_existence_required)
2070 0.005 0 2.506 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:86(generate)
2070 0.002 0 2.501 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:109(generate)
2070 0.003 0 2.499 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:94(generate_now)
2070 0.01 0 2.496 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:65(get_state)
690 0.001 0 2.292 0.003 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:148(__nonzero__)
690 0.005 0 2.291 0.003 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:124(__bool__)
2070 0.007 0 2.276 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:112(_exists)
2070 0.01 0 2.269 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/storages/backends/s3boto.py:409(exists)
4140 0.004 0 2.14 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/storages/backends/s3boto.py:282(entries)
1633 0.003 0 2.135 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/storages/backends/s3boto.py:288()
1633 0.001 0 2.129 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/bucketlistresultset.py:24(bucket_lister)
2 0 0 2.128 1.064 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/bucket.py:390(_get_all)
2 0 0 2.128 1.064 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/bucket.py:426(get_all_keys)
1331 0.003 0 1.288 0.001 /usr/lib/python2.7/ssl.py:335(recv)
1331 1.285 0.001 1.285 0.001 /usr/lib/python2.7/ssl.py:254(read)
2 0 0 0.983 0.491 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/connection.py:886(_mexe)
2 0 0 0.983 0.491 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/connection.py:643(make_request)
2 0 0 0.983 0.491 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/connection.py:1062(make_request)
2 0.004 0.002 0.896 0.448 /usr/lib/python2.7/httplib.py:585(_read_chunked)
2 0 0 0.896 0.448 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/connection.py:393(read)
2 0 0 0.896 0.448 /usr/lib/python2.7/httplib.py:540(read)
166 0.002 0 0.777 0.005 /usr/lib/python2.7/httplib.py:643(_safe_read)
166 0.005 0 0.775 0.005 /usr/lib/python2.7/socket.py:336(read)
2 0 0 0.568 0.284 /usr/lib/python2.7/httplib.py:793(send)
2 0 0 0.568 0.284 /usr/lib/python2.7/httplib.py:998(_send_request)
2 0 0 0.568 0.284 /usr/lib/python2.7/httplib.py:820(_send_output)
2 0 0 0.568 0.284 /usr/lib/python2.7/httplib.py:977(request)
2 0 0 0.568 0.284 /usr/lib/python2.7/httplib.py:962(endheaders)
1 0 0 0.567 0.567 /usr/lib/python2.7/httplib.py:1174(connect)
1380 0.001 0 0.547 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:82(url)
1380 0.007 0 0.546 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:72(_storage_attr)
105 0.009 0 0.528 0.005 /usr/lib/python2.7/socket.py:406(readline)
2 0 0 0.413 0.207 /usr/lib/python2.7/httplib.py:408(begin)
2 0 0 0.413 0.207 /usr/lib/python2.7/httplib.py:1015(getresponse)
2 0 0 0.407 0.203 /usr/lib/python2.7/httplib.py:369(_read_status)
2750 0.003 0 0.337 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/fields.py:399(get_attribute)
1 0.223 0.223 0.335 0.335 /usr/lib/python2.7/socket.py:537(create_connection)
2865 0.012 0 0.334 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/fields.py:65(get_attribute)
1610 0.005 0 0.314 0 /home/my_app/.virtualenvs/my_app/src/django-s3-folder-storage/s3_folder_storage/s3.py:13(url)
1610 0.012 0 0.309 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/storages/backends/s3boto.py:457(url)
690 0.005 0 0.292 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/models/fields/utils.py:10(__get__)
690 0.007 0 0.251 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:20(__init__)
2 0 0 0.248 0.124
>>>> cutting here, low impact calls
I am in the process of writing a utility to convert a mips instruction into its hex (4 bytes) format. Everything was going ok with instructions such as ADD, etc. But with mthi and mtlo, there is a different in the output I am getting compare to what is expected. I am not sure what exact version of mips this is.
Here is what I am getting:
mthi $t2 = 01400011
mthi $s0 = 02000011
mtlo $t8 = 03000013
mtlo $a3 = 00e00013
Here is what I am getting:
mthi $t2
1010 00000 00000 00000 010001
Does anyone know how the inner bits are being calculated and what version of mips that is? Thanks.
Information from "See MIPS Run", for all MIPS ISA it covers (up through MIPS IV) there is no difference in the MIPS32 instruction.
This is the format for mfhi, mtlo, mfhi, mtlo (mfhi/mflo included for comparison and completeness)
bit: 31-26 25-21 20-16 15-11 10-6 5-0
mfhi rd 0 0 0 rd 0 16
mthi rs 0 rs 0 0 0 17
mflo rd 0 0 0 rd 0 18
mtlo rs 0 rs 0 0 0 19
So mthi $t2 (register #10)
mthi $t2 000000 01010 00000 00000 00000 01001
The MIPS 32R2 encoding of MTHI rs is binary 000000 sssss 000 0000 0000 0000 010001.
So MTHI $t2 is binary 000000 01010 000 0000 0000 0000 010001