Adam optimizer has flaws when used with weight decay. In 2018, AdamW optimizer has been proposed.
Is there any standard way to implement AdamW in MXNet framework (python implementation)? There is mxnet.optimizer.Adam class, but no mxnet.optimizer.AdamW one (checked in mxnet-cu102==1.6.0, mxnet==1.5.0 package versions).
P.S. I asked this questions on MXNet forum and on datascience.stackexchange.com, but to no avail.
Short answer: There isn't a standard way to use AdamW in Gluon yet, but there is some existing work in that direction that would make that relatively easy to add.
Longer answer:
People have been asking for this feature - a lot :) See: https://github.com/apache/incubator-mxnet/issues/9182
Gluon-NLP has a working version of AdamW - possibly slightly different from the one in the original paper: https://github.com/eric-haibin-lin/gluon-nlp/blob/df63e2c2a4d6b998289c25a38ffec8f4ff647ff4/src/gluonnlp/optimizer/bert_adam.py
The adamw_update() operator was added with this pull request: https://github.com/apache/incubator-mxnet/pull/13728 This is first released in MXNet 1.6.0.
Unfortunately, it looks like there isn't a way to use this with gluon.Trainer directly right now, without copying/modifying the BERTAdam code (or writing something similar from scratch). That would be a very nice thing to add to Gluon.
Please let me know if you get this working, as I'd love to be able to use that as well.
Related
I try to code floating adder;
https://github.com/ElectronNest/FPU/blob/master/FloatAdd.scala
This is half way.
The normalization is huge code part, so I would like to use for-loop or some equivalent representation method.
Is it possible to use loop or we need strict coding?
Best,
S.Takano
This is a very general and large question. The equivalent of a for loop in hardware can be implemented using a number of techniques, pretty much all of them involving registers to hold state information. Looking at your code I would suggest that you start a little smaller and work on syntax, I see many syntax errors currently. I use IntelliJ community edition as an editor because it does a great job with helping to get the code properly structured. I also would strongly recommend starting from the chisel-template repository. It has the proper layout and examples of a working circuit and unit testing harness. Then start with a smaller implementation that does something simple like just pass input to output and runs in a test harness, then slowly build up the circuit to achieve your goals.
Good luck!
Welcome and thank you for your interest in Chisel!
I would like to echo Chick's suggestion to start from something small that compiles and simulates and build up from there. In particular, the linked code above conflates some Scala vs. Chisel constructs (eg. Scala's if else, vs. Chisel's when, .elsewhen, .otherwise), as well as some Verilog vs. Chisel concepts (eg. bit indexing with [high:low] vs. Chisel's (high, low))
In case you haven't seen it, I would suggest taking a look at the Chisel Bootcamp which helps explain how to use constructs like for loops to generate hardware.
I'll also plug my own responses to this question on the chisel-users mailing list where I tried to explain some of the intuition behind writing Chisel generators, including differentiating if and when and using for loops.
since Apache released the first final version of Johnzon, it would be really interesting to see if there are already some comparison between Johnzon and FastXML Jackson to see if it is worth to switch. The most important topic is probably the performance.
Has anyone already done performance tests? Can you share your result?
Best
There are some performance benchmarks up on github.
But for each of them you really have to verify if the benchmark is actually correctly implemented.
For what I've seen most benchmarks use the official javax.* APIs in a sub-optimal way. Most use Json.createGenerator, etc but they should actually use JsonProvider.provider() and store this away for your operations. Then call createGenerator etc on this JsonProvider.
That way you can make sure that you really get comparable results.
We have done quite a few tests and for me the numbers of Johnzon look really good. And especially since it's much smaller than most other JSON libs.
As mentioned in several other sources and mailing lists(TomEE, for example), the performance gain, if any, is negligible especially when you compare it to the overall request-response processing chain.
If you use Spring Boot, you will find a lot more community support and flexibility in terms of features for Jackson.
Jackson has tons of different modules and good support for other JVM languages(for example KotlinModule).
We, in my project, also use quite a lot of Clojure, where we use Cheshire, which relies on Jackson under the hood.
In the end, it's up to you what to use and whether the cases I mentioned are applicable to your project, but so far I haven't seen any compelling performance reports about Johnson and until it happens, I would go for a library with a lot higher adoption in the industry.
I'm working on a CRAM auth system using a Flash/Flex client and a Java server (Red5). I have used the as3crypto library before, but as far as I know it does not support PBKDF2. This algorithm is suggested for password encryption by NIST so its what I want to use. Does anyone know of an AS3 compatible library with this algorithm, specifically PBKDF2WithHmacSHA1?
Refs:
http://code.google.com/p/as3crypto/
http://csrc.nist.gov/publications/nistpubs/800-132/nist-sp800-132.pdf
This one seems OK: http://code.google.com/p/as3-pbkdf2
Did you try putting 'as3 PBKDF2' into google? Took me all of 10 seconds to find these two as they're the top two answers.
The first would suit your needs, but the second seems more versatile as you can specify a hash function.
http://code.google.com/p/as3-pbkdf2/
http://code.google.com/p/as3-pbkdf2-lib/
I'm wrapping a C++ library in PHP using SWIG and there have been some occasions where I want to modify the generated code (both generated C++ and PHP):
Fix code-generation errors
Add code that makes sense in PHP, but not in C++ (e.g. type checking)
Add documentation tags (e.g. phpDoc)
I'm currently automating these modifications with patch. This approach works, but it seems high-maintenance and fragile. Is there a better way of doing this?
The best bet is to have your code generator generate correct code for your needs. Hand-tweaking generated output is unsustainable. You'll have to tweak it again any time the input changes.
If a tool is producing flatly erroneous output, it's ideal to repair it and submit patches back to the maintainer. If the output is correct for some circumstances but wrong for yours, I'd suggest to add an option that changes the behavior to what you need.
Sometimes, you can use a short program that automatically does an intelligent job of patching your generated code, so that you don't need a manual process to make patches.
Alternatively, you could write your own code generator, but I suspect that's much more work than you want. It also depends on what you're doing. Sometimes code-generation is really just macro-expansion, and there are plenty of good tools for that in the wild.
Good luck!
You may end up having a maintenance nightmare later on. Instead of SWIG you might consider using another generative approach that:
Let you add your custom code directly on the model (so that you won't need to add it post-generation)
Let you define your own generator. This feature alone could take out the need to add custom code all along.
The problem of using third-party generators is that they never really generate what you want. The problem of writing your own code generators is that it's much more work. You choose.
But correcting an automation with another automation...
Code generation is quite a wide topic and there are definitely many other approaches, which might be more interresting to you as mentioned above.
But if you do not want to use other tool, depending on what code is generated and on the PHP OO capabilities, you might use the Generation Gap pattern.
How does one study open-source libraries code, particularly standard libraries?
The code base is often vast and hard to navigate. How to find some function or class definition?
Do I search through downloaded source files?
Do I need cvs/svn for that?
Maybe web-search?
Should I just know the structure of the standard library?
Is there any reference on it?
Or do some IDEs have such features? Or some other tools?
How to do it effectively without one?
What are the best practices of doing this in any open-source libraries?
Is there any convention of how are sources manipulated on Linux/Unix systems?
What are the differences for specific programming languages?
Broad presentation of the subject is highly encouraged.
I mark this 'community wiki' so everyone can rephrase and expand my awkward formulations!
Update: Probably didn't express the problem clear enough. What I want to, is to view just the source code of some specific library class or function. And the problem is mostly about work organization and usability - how do I navigate in the huge pile of sources to find the thing, maybe there are specific tools or approaches? It feels like there should've long existed some solution(s) for that.
One thing to note is that standard libraries are sometimes (often?) optimized more than is good for most production code.
Because they are widely used, they have to perform well over a wide variety of conditions, and may be full of clever tricks and special logic for corner cases.
Maybe they are not the best thing to study as a beginner.
Just a thought.
Well, I think that it's insane to just site down and read a library's code. My approach is to search whenever I come across the need to implement something by myself and then study the way that it's implemented in those libraries.
And there's also allot of projects/libraries with excellent documentation, which I find more important to read than the code. In Unix based systems you often find valuable information in the man pages.
Wow, that's a big question.
The short answer: it depends.
The long answer:
Some libraries provide documentation while others don't. Standard libraries are usually pretty well documented, whether your chosen implementation of the library includes documentation or not. For instance you may have found an implementation of the c standard library without documentation but the c standard has been around long enough that there are hundreds of good reference books available. Documentation with hyperlinks is a very useful way to learn a new API. In any case the first place I would look is the library's main website
For less well known libraries lacking documentation I find two different approaches very helpful.
First is a doc generator. Nearly every language I know of has one. It basically parses an source tree and creates documentation (usually as html or xml) which can be used to learn a library. Some use specially formatted comments in the code to create more complete documentation. JavaDoc is one good example of this. Doc generators for many other languages borrow from JavaDoc.
Second an IDE with a class browser. These act as a sort of on the fly documentation. Some display just the library's interface. Other's include description comments from the library's source.
Both of these will require access to the libraries source (which will come in handy if you intend actually use a library).
Many of these tools and techniques work equally well for closed/proprietary libraries.
The standard Java libraries' source code is available. For a beginning Java programmer these can be a great read. Especially the Collections framework is a good place to start. Take for instance the implementation of ArrayList and learn how you can implement a resizeable array in Java. Most of the source has even useful comments.
The best parts to read are probably whose purpose you can understand immediately. Start with the easy pieces and try to follow all the steps that are hidden behind that single call you make from your own code.
Something I do from time to time :
apt-get source foo
Then new C++ project (or whatever) in Eclipse and import.
=> Wow ! Browsable ! (use F3)