I've been trying to resolve this problem for a while now, I even gave it a shot to rewrite the entire program but without success. The application is running on VueJS 2.3.3 and is supposed to be running on Chromium in combination with a Raspberry Pi (irrelevant information, for now).
We're working with several components which are being included in a single file, later on this file will be compiled using either gulp or npm run dev. When the instance of VueJS initializes, a request will be send using Vue Resource's $http option. This'll receive a json response with a size of around 30mb. This'll be saved in the data array, as seen here:
this.$http.get('<url>' + this.token)
.then((response) => {
this.properties = response.properties;
});
This data will later on be used for further actions, another thing that is worth mentioning is that the data is being refreshed every once in a while. Which is where I think the problem occurs, if I'm not refreshing the data every 5 minutes (can be longer too, really depends on the way I'm testing) the program runs fine. It's just that I want to refresh the data every once in a while to retrieve new information. The way of setting a timeout which I'm using is as following:
this.dataTimeout = setTimeout(this.refreshData, 300000);
Each (so called) property has at least 6 base64 images saved in it's JSON, which are later used to present to the user. Besides that, there is a name, address, and some other tiny bits of data. It doesn't sound all that wrong but I'm getting the feeling that each response makes the memory grow so intense that even a desktop is getting trouble running it.
Each 10 seconds a new property will be presented on the user's screen including the images, street, location, etc. I'm not sure if there is a memory leak in my code or if I'm forgetting something. A few questions pop up in my head:
Do I need to reset the response I'm getting from the server back to
null or undefined?
Could there be a leak in one of the plugins I'm using (Just VueResources)?
For how long can a VueJS instance stay alive, is there any recommended time to reload the entire program?
What thinks should I take in consideration in order to achieve this at all?
I've taken out the data renewal and put a demo project online, this can be seen right here. The main problem I'm having is that the browser just runs out of memory and shows us the amazing Aw snap! page from Chrome. I tried taking snapshots from the memory usage but it all seems fine, it just explodes randomly after a while.
Well, I don't know what really does your app, but are your 30Mb of data really useful / essential ? In JSON moreover ?
Maybe you don't need all this data, and you could just adapt the data to your needs. For example, keep your JSON store data, and retrieve your Base64 images by another way.
I don't understand why you store in memory images. Images are just useful for display purpose in my opinion.
So I think 30Mb is really huge. But maybe I'm wrong ?
By the way, I've tested with Firefox Nightly, no problem here. Doesn't seem to crash. Maybe I don't encounter the refresh call ?
In Magento we had a problem with extremely fast changing stocklevels & deliverytimes of products originating from a variety of suppliers.
We have over 400k products in catalogue, and about 50-150k changes in stocklevels every day, with popular products having hourly changes - there was no way Magento could keep up with this in a timely manner.
We solved most of this problem using direct sql in the product list (list.phtml), view (view.phtml) and in the shoppingcart.
This extremely simple query accesses - if needed - the table in our DB that holds the stocklevels, productstatus and deliverytimes for each product and displays this next to the product to assure we display accurate stock & deliverytime information to our custmers.
query:
$query = "SELECT external_stock, productstatus, deliveryperiod FROM products_supplierstatus WHERE sku='$sku'";
$data = Mage::getSingleton('core/resource')->getConnection('core_read')->fetchAll($query);
With php we loop through the status & stocklevels and depending a few factors we display a tekst that informs the customer about the expected deliverytime of this product.
So far, so good - Always worked like a charm and solved our problem.
However, we decided to combine these routine in a simple module, so we can always call the same block, and we would be able to manage changes for deliverytimes more easily (in 1 place instead of 3).
Sounded easy enough so we created a small module and called this in the list & view pages:
<?php echo $this->getLayout()->createBlock('dynamic/delivery')->setData('prodid', $_product->getId())->setTemplate('dvdo/showdelivery.phtml')->toHtml(); ?>
also works great.. but: mainly noticeable on the list view the loading times of the pages is much longer than it used to be.
We hardly changed anything in the query or in the php that loops through the results and creates the html.
Because we need near-realtime stockinformation we do not cache these blocks.
What's the reason behind this?
Does it take so much extra resources in Magento to build this block compared to executing nearly the same code in the phtml file?
If anyone can point me in the right direction to get this module working properly, I would be extremely gratefull!
There's too many factors for anyone to guess at the right answer. You'll need to profile and debug this yourself. Here's some guidance.
It may not be what you think. Magento's a huge application, with many layers of abstraction that interact with each other, with so many different interactions that's it's often difficult to keep things straight. You may have changed something else that's impacting performance. This is why you need to debug and profile
An intro to Varien_Profiler, the profiling class that ships with Magento
If you completely remove the contents of dvdo/showdelivery.phtml so it's a blank file, does the performance problem still happen?
Is this being rendered more times than you think it is? Is the block inside a loop? (some logging in our template can help reveal this)
What, exactly, is in your template file? Your SQL query uses a SKU, but your block (presumably) uses the a product ID
Speaking of product ID, where is the $_product in $_product->getId() coming from?
What does your block class for dynamic/delivery look like? Are there heavy operations in the PHP constructor? Magento's _construct method? The prepareLayout method?
It's possible, although I doubt it, that the extra work of instantiating an additional class (your block), and loading another template file (the phtml), buts your particular system over some *nix* threshold for too many PHP classes or files opened at once. This is more likely if your block instantiation is in a loop.
If may be your original SQL was something that you database system (MySQL?) could easily apply SQL caching to, but that whatever query you have in dvdo/showdelivery.phtml is not something it can easily cache on the SQL level.
If you figure it out, be sure to update the comments — I'd be interested in hearing what it eventually was. Good luck!
I recommend you replace $this->getLayout()->createBlock('dynamic/delivery') with block created via .xml by placing nodes and values in proper handle.
I am not sure if these blocks are conditional.Unless they are conditional blocks for faster implementation using XML file for block creation will be faster.
I tried to look for free H.264 editors that could do such forementioned tasks, but i couldn't find any suitable results (probably due to my incorrect(?) search terms).
Basically, i have quite a few 20 to 40 second looping movie files (rendered with Adobe Premiere), and i would like to multiply an individual movie to about ten or twenty fold, and then save it again, preferably without the need to re-render (or re-encode?). Is this possible?
Hopefully i managed to make myself understood, thanks :)
After a whole bunch of searching and testing, i ended up with Avidemux and it did the task perfectly, this time around. I'm sure there would've been a way to automatize the appending / "multiplying" of the file, but with a tablet pen (main click set as double click) and specific folder it truly went by like a breeze.
I was also interested of Smart Cutter (i quite enjoyed the interface after getting used to it), but since it was trialware the choice was clear. Still, if i would have to do these kind of tasks more often, i might consider purchasing it.
I read some website development materials on the Web and every time a person is asking for the organization of a website's js, css, html and php files, people suggest single js for the whole website. And the argument is the speed.
I clearly understand the fewer request there is, the faster the page is responded. But I never understand the single js argument. Suppose you have 10 webpages and each webpage needs a js function to manipulate the dom objects on it. Putting 10 functions in a single js and let that js execute on every single webpage, 9 out of 10 functions are doing useless work. There is CPU time wasting on searching for non-existing dom objects.
I know that CPU time on individual client machine is very trivial comparing to bandwidth on single server machine. I am not saying that you should have many js files on a single webpage. But I don't see anything go wrong if every webpage refers to 1 to 3 js files and those js files are cached in client machine. There are many good ways to do caching. For example, you can use expire date or you can include version number in your js file name. Comparing to mess the functionality in a big js file for all needs of many webpages of a website, I far more prefer split js code into smaller files.
Any criticism/agreement on my argument? Am I wrong? Thank you for your suggestion.
A function does 0 work unless called. So 9 empty functions are 0 work, just a little exact space.
A client only has to make 1 request to download 1 big JS file, then it is cached on every other page load. Less work than making a small request on every single page.
I'll give you the answer I always give: it depends.
Combining everything into one file has many great benefits, including:
less network traffic - you might be retrieving one file, but you're sending/receiving multiple packets and each transaction has a series of SYN, SYN-ACK, and ACK messages sent across TCP. A large majority of the transfer time is establishing the session and there is a lot of overhead in the packet headers.
one location/manageability - although you may only have a few files, it's easy for functions (and class objects) to grow between versions. When you do the multiple file approach sometimes functions from one file call functions/objects from another file (ex. ajax in one file, then arithmetic functions in another - your arithmetic functions might grow to need to call the ajax and have a certain variable type returned). What ends up happening is that your set of files needs to be seen as one version, rather than each file being it's own version. Things get hairy down the road if you don't have good management in place and it's easy to fall out of line with Javascript files, which are always changing. Having one file makes it easy to manage the version between each of your pages across your (1 to many) websites.
Other topics to consider:
dormant code - you might think that the uncalled functions are potentially reducing performance by taking up space in memory and you'd be right, however this performance is so so so so minuscule, that it doesn't matter. Functions are indexed in memory and while the index table may increase, it's super trivial when dealing with small projects, especially given the hardware today.
memory leaks - this is probably the largest reason why you wouldn't want to combine all the code, however this is such a small issue given the amount of memory in systems today and the better garbage collection browsers have. Also, this is something that you, as a programmer, have the ability to control. Quality code leads to less problems like this.
Why it depends?
While it's easy to say throw all your code into one file, that would be wrong. It depends on how large your code is, how many functions, who maintains it, etc. Surely you wouldn't pack your locally written functions into the JQuery package and you may have different programmers that maintain different blocks of code - it depends on your setup.
It also depends on size. Some programmers embed the encoded images as ASCII in their files to reduce the number of files sent. These can bloat files. Surely you don't want to package everything into 1 50MB file. Especially if there are core functions that are needed for the page to load.
So to bring my response to a close, we'd need more information about your setup because it depends. Surely 3 files is acceptable regardless of size, combining where you would see fit. It probably wouldn't really hurt network traffic, but 50 files is more unreasonable. I use the hand rule (no more than 5), but surely you'll see a benefit combining those 5 1KB files into 1 5KB file.
Two reasons that I can think of:
Less network latency. Each .js requires another request/response to the server it's downloaded from.
More bytes on the wire and more memory. If it's a single file you can strip out unnecessary characters and minify the whole thing.
The Javascript should be designed so that the extra functions don't execute at all unless they're needed.
For example, you can define a set of functions in your script but only call them in (very short) inline <script> blocks in the pages themselves.
My line of thought is that you have less requests. When you make request in the header of the page it stalls the output of the rest of the page. The user agent cannot render the rest of the page until the javascript files have been obtained. Also javascript files download sycronously, they queue up instead of pull at once (at least that is the theory).
Stacker Nobody asked about the most shocking thing new programmers find as they enter the field.
Very high on the list, is the impact of inheriting a codebase with which one must rapidly become acquainted. It can be quite a shock to suddenly find yourself charged with maintaining N lines of code that has been clobbered together for who knows how long, and to have a short time in which to start contributing to it.
How do you efficiently absorb all this new data? What eases this transition? Is the only real solution to have already contributed to enough open-source projects that the shock wears off?
This also applies to veteran programmers. What techniques do you use to ease the transition into a new codebase?
I added the Community-Building tag to this because I'd also like to hear some war-stories about these transitions. Feel free to share how you handled a particularly stressful learning curve.
Pencil & Notebook ( don't get distracted trying to create a unrequested solution)
Make notes as you go and take an hour every monday to read thru and arrange the notes from previous weeks
with large codebases first impressions can be deceiving and issues tend to rearrange themselves rapidly while you are familiarizing yourself.
Remember the issues from your last work environment aren't necessarily valid or germane in your new environment. Beware of preconceived notions.
The notes/observations you make will help you learn quickly what questions to ask and of whom.
Hopefully you've been gathering the names of all the official (and unofficial) stakeholders.
One of the best ways to familiarize yourself with inherited code is to get your hands dirty. Start with fixing a few simple bugs and work your way into more complex ones. That will warm you up to the code better than trying to systematically review the code.
If there's a requirements or functional specification document (which is hopefully up-to-date), you must read it.
If there's a high-level or detailed design document (which is hopefully up-to-date), you probably should read it.
Another good way is to arrange a "transfer of information" session with the people who are familiar with the code, where they provide a presentation of the high level design and also do a walk-through of important/tricky parts of the code.
Write unit tests. You'll find the warts quicker, and you'll be more confident when the time comes to change the code.
Try to understand the business logic behind the code. Once you know why the code was written in the first place and what it is supposed to do, you can start reading through it, or as someone said, prolly fixing a few bugs here and there
My steps would be:
1.) Setup a source insight( or any good source code browser you use) workspace/project with all the source, header files, in the code base. Browsly at a higher level from the top most function(main) to lowermost function. During this code browsing, keep making notes on a paper/or a word document tracing the flow of the function calls. Do not get into function implementation nitti-gritties in this step, keep that for a later iterations. In this step keep track of what arguments are passed on to functions, return values, how the arguments that are passed to functions are initialized how the value of those arguments set modified, how the return values are used ?
2.) After one iteration of step 1.) after which you have some level of code and data structures used in the code base, setup a MSVC (or any other relevant compiler project according to the programming language of the code base), compile the code, execute with a valid test case, and single step through the code again from main till the last level of function. In between the function calls keep moting the values of variables passed, returned, various code paths taken, various code paths avoided, etc.
3.) Keep repeating 1.) and 2.) in iteratively till you are comfortable up to a point that you can change some code/add some code/find a bug in exisitng code/fix the bug!
-AD
I don't know about this being "the best way", but something I did at a recent job was to write a code spider/parser (in Ruby) that went through and built a call tree (and a reverse call tree) which I could later query. This was slightly non-trivial because we had PHP which called Perl which called SQL functions/procedures. Any other code-crawling tools would help in a similar fashion (i.e. javadoc, rdoc, perldoc, Doxygen etc.).
Reading any unit tests or specs can be quite enlightening.
Documenting things helps (either for yourself, or for other teammates, current and future). Read any existing documentation.
Of course, don't underestimate the power of simply asking a fellow teammate (or your boss!) questions. Early on, I asked as often as necessary "do we have a function/script/foo that does X?"
Go over the core libraries and read the function declarations. If it's C/C++, this means only the headers. Document whatever you don't understand.
The last time I did this, one of the comments I inserted was "This class is never used".
Do try to understand the code by fixing bugs in it. Do correct or maintain documentation. Don't modify comments in the code itself, that risks introducing new bugs.
In our line of work, generally speaking we do no changes to production code without good reason. This includes cosmetic changes; even these can introduce bugs.
No matter how disgusting a section of code seems, don't be tempted to rewrite it unless you have a bugfix or other change to do. If you spot a bug (or possible bug) when reading the code trying to learn it, record the bug for later triage, but don't attempt to fix it.
Another Procedure...
After reading Andy Hunt's "Pragmatic Thinking and Learning - Refactor Your Wetware" (which doesn't address this directly), I picked up a few tips that may be worth mentioning:
Observe Behavior:
If there's a UI, all the better. Use the app and get a mental map of relationships (e.g. links, modals, etc). Look at HTTP request if it helps, but don't put too much emphasis on it -- you just want a light, friendly acquaintance with app.
Acknowledge the Folder Structure:
Once again, this is light. Just see what belongs where, and hope that the structure is semantic enough -- you can always get some top-level information from here.
Analyze Call-Stacks, Top-Down:
Go through and list on paper or some other medium, but try not to type it -- this gets different parts of your brain engaged (build it out of Legos if you have to) -- function-calls, Objects, and variables that are closest to top-level first. Look at constants and modules, make sure you don't dive into fine-grained features if you can help it.
MindMap It!:
Maybe the most important step. Create a very rough draft mapping of your current understanding of the code. Make sure you run through the mindmap quickly. This allows an even spread of different parts of your brain to (mostly R-Mode) to have a say in the map.
Create clouds, boxes, etc. Wherever you initially think they should go on the paper. Feel free to denote boxes with syntactic symbols (e.g. 'F'-Function, 'f'-closure, 'C'-Constant, 'V'-Global Var, 'v'-low-level var, etc). Use arrows: Incoming array for arguments, Outgoing for returns, or what comes more naturally to you.
Start drawing connections to denote relationships. Its ok if it looks messy - this is a first draft.
Make a quick rough revision. Its its too hard to read, do another quick organization of it, but don't do more than one revision.
Open the Debugger:
Validate or invalidate any notions you had after the mapping. Track variables, arguments, returns, etc.
Track HTTP requests etc to get an idea of where the data is coming from. Look at the headers themselves but don't dive into the details of the request body.
MindMap Again!:
Now you should have a decent idea of most of the top-level functionality.
Create a new MindMap that has anything you missed in the first one. You can take more time with this one and even add some relatively small details -- but don't be afraid of what previous notions they may conflict with.
Compare this map with your last one and eliminate any question you had before, jot down new questions, and jot down conflicting perspectives.
Revise this map if its too hazy. Revise as much as you want, but keep revisions to a minimum.
Pretend Its Not Code:
If you can put it into mechanical terms, do so. The most important part of this is to come up with a metaphor for the app's behavior and/or smaller parts of the code. Think of ridiculous things, seriously. If it was an animal, a monster, a star, a robot. What kind would it be. If it was in Star Trek, what would they use it for. Think of many things to weigh it against.
Synthesis over Analysis:
Now you want to see not 'what' but 'how'. Any low-level parts that through you for a loop could be taken out and put into a sterile environment (you control its inputs). What sort of outputs are you getting. Is the system more complex than you originally thought? Simpler? Does it need improvements?
Contribute Something, Dude!:
Write a test, fix a bug, comment it, abstract it. You should have enough ability to start making minor contributions and FAILING IS OK :)! Note on any changes you made in commits, chat, email. If you did something dastardly, you guys can catch it before it goes to production -- if something is wrong, its a great way to get a teammate to clear things up for you. Usually listening to a teammate talk will clear a lot up that made your MindMaps clash.
In a nutshell, the most important thing to do is use a top-down fashion of getting as many different parts of your brain engaged as possible. It may even help to close your laptop and face your seat out the window if possible. Studies have shown that enforcing a deadline creates a "Pressure Hangover" for ~2.5 days after the deadline, which is why deadlines are often best to have on a Friday. So, BE RELAXED, THERE'S NO TIMECRUNCH, AND NOW PROVIDE YOURSELF WITH AN ENVIRONMENT THAT'S SAFE TO FAIL IN. Most of this can be fairly rushed through until you get down to details. Make sure that you don't bypass understanding of high-level topics.
Hope this helps you as well :)
All really good answers here. Just wanted to add few more things:
One can pair architectural understanding with flash cards and re-visiting those can solidify understanding. I find questions such as "Which part of code does X functionality ?", where X could be a useful functionality in your code base.
I also like to open a buffer in emacs and start re-writing some parts of the code base that I want to familiarize myself with and add my own comments etc.
One thing vi and emacs users can do is use tags. Tags are contained in a file ( usually called TAGS ). You generate one or more tags files by a command ( etags for emacs vtags for vi ). Then we you edit source code and you see a confusing function or variable you load the tags file and it will take you to where the function is declared ( not perfect by good enough ). I've actually written some macros that let you navigate source using Alt-cursor,
sort of like popd and pushd in many flavors of UNIX.
BubbaT
The first thing I do before going down into code is to use the application (as several different users, if necessary) to understand all the functionalities and see how they connect (how information flows inside the application).
After that I examine the framework in which the application was built, so that I can make a direct relationship between all the interfaces I have just seen with some View or UI code.
Then I look at the database and any database commands handling layer (if applicable), to understand how that information (which users manipulate) is stored and how it goes to and comes from the application
Finally, after learning where data comes from and how it is displayed I look at the business logic layer to see how data gets transformed.
I believe every application architecture can de divided like this and knowning the overall function (a who is who in your application) might be beneficial before really debugging it or adding new stuff - that is, if you have enough time to do so.
And yes, it also helps a lot to talk with someone who developed the current version of the software. However, if he/she is going to leave the company soon, keep a note on his/her wish list (what they wanted to do for the project but were unable to because of budget contraints).
create documentation for each thing you figured out from the codebase.
find out how it works by exprimentation - changing a few lines here and there and see what happens.
use geany as it speeds up the searching of commonly used variables and functions in the program and adds it to autocomplete.
find out if you can contact the orignal developers of the code base, through facebook or through googling for them.
find out the original purpose of the code and see if the code still fits that purpose or should be rewritten from scratch, in fulfillment of the intended purpose.
find out what frameworks did the code use, what editors did they use to produce the code.
the easiest way to deduce how a code works is by actually replicating how a certain part would have been done by you and rechecking the code if there is such a part.
it's reverse engineering - figuring out something by just trying to reengineer the solution.
most computer programmers have experience in coding, and there are certain patterns that you could look up if that's present in the code.
there are two types of code, object oriented and structurally oriented.
if you know how to do both, you're good to go, but if you aren't familiar with one or the other, you'd have to relearn how to program in that fashion to understand why it was coded that way.
in objected oriented code, you can easily create diagrams documenting the behaviors and methods of each object class.
if it's structurally oriented, meaning by function, create a functions list documenting what each function does and where it appears in the code..
i haven't done either of the above myself, as i'm a web developer it is relatively easy to figure out starting from index.php to the rest of the other pages how something works.
goodluck.