I am having a very large JSON file, but it's poorly formatted so that all the chars are in one line only, no \n anywhere, which makes it difficult to read and understand. And since it's more than one kilobyte file, editing it manually is out of the question.
I am looking for a command or some other way to format a JSON file quickly, for human readability, and save it into the same file or another one. Ideally I shouldn't have to setup too many things. I know that some IDEs include automatic formatting features, however installing another IDE onto my work computer is not an option for me.
The JSON is in a file? You could do this:
python -m json.tool my_json.json
Built into python from version 2.5? 2.6? And it will pretty-ify the json for you.
Related
I need to convert many files from .lwo format to .obj or .stl. I have too many to convert "by hand", meaning I don't want to use online tools or import/export the files one by one in Blender or similar.
So I'm trying to do so with a program that would load up each file, convert, then save a new stl . The files are numbered "file000001", "file000002", etc. to make importing easier.
Is there any program out there that will do this? If not, how would I go about accomplishing my goal?
As far as languages go, I am most effective with Processing/Java. I found this which might be similar but doesn't relate to LWOs.
Thanks for any help.
I just found assimp which has a command line tool to convert different file types. Thanks everyone who answered!
I'm sure you can find a few editors that import .lwo and export .obj
For example, Wings3D does that and free/opensource/lightweight.
Wings is scriptable using erlang.
Blender has LWO importer too, but it's not enabled by default. you need to go to Preferences > Addons and enable it there:
Blender has a Python API which should be easy to pickup.
This would allow you to write a script that does a batch conversion (reads a directory, traverses files, imports .lwo, transforms (scales/rotates if needed), exports .obj)
Perhaps if you search enough maybe there is a 3d file format batch converter already out there and .lwo/.obj are old enough formats so might be likely to be supported.
If you want to implement something from scratch, you need to look into each file format (e.g. lightwave object, obj ) to be able to parse and export.
Hopefully there's a java library that for you. I'd start with a 3D java game engine. For example here's a java .LWO importer found via JMonkey.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
I would like to hear your suggestions on how to handle a large (40MB) JSON file on Ubuntu. I would like to see it pretty printed in vim or gedit or any other editor. One can find numerious tutorials on how to prettify the JSON, however, they do not have to deal with large input. I also imagine I could pipe the data through pygments or any other syntax highlighter. I am curious to hearing your ideas.
Example download:
wget -O large-dataset.json http://data.wien.gv.at/daten/wfs?service=WFS&request=GetFeature&version=1.1.0&typeName=ogdwien:BAUMOGD&srsName=EPSG:4326&outputFormat=json
Please mind the download size!
EDIT: I found out that meld is working somewhat. The application does not load the whole file at once which would block the user interface. Instead it sequentially reads the file content.
Python JSON's module can do this too (python -m json.tool), e.g.:
cat myjsonfile.json | python -m json.tool > pretty.json
If you just want to visualize (and search) a json file, Firefox does a pretty good job. I don't have a 40MB file on hand, but it easily handled a 9MB one.
Just drag the JSON file to Firefox, or run:
firefox your_file.json
jq is a lightweight commandline JSON processor and works well!
for this you need to install jq with the command below(if you are using apt package manager).
sudo apt-get install jq
Below command will pretty json to a new file.
jq '.' non-pretty.json > pretty.json
Also we can filter the json with jq, I found it to be very helpful while working with large geojson files, for instance below command will save only properties of first feature.
jq '.features[0].properties' geojson_file.json > pretty.json
Hope this will be helpful!
Do you have KDE or any other visual environment? If yes, have you tried using the chrome extension JSONView?
I usually use Sublime Text for this purpose. There is a dedicated plugin for this job.
The plugin Pretty JSON parses the JSON contents selected, and prints them in a structured way.
All you need to do is to select the contents and press Ctrl+Alt+j.
The core usage is pretty formatting large json. I tested Chrome extension JSON View with 25MB json file. It crashes on loading this as a local file or from network. By crash, I mean JSON will not get formatted and on looking into JSON view options, you will get a crash message. I also tried similar addons for firefox. I tried online json formatters as well.
Found this library - jsonpps. Works pretty well to pretty format large json from command line, taking input and saving the formatted json as separate file. It can also save in the same file (need optional parameter)
One drawback, To install and run, one should be familier with Java and Maven.
To install & run:
git clone https://github.com/bazaarvoice/jsonpps.git
mvn clean package
cd target
java -jar jsonpps-1.2-SNAPSHOT.jar -o /path/to/output.json /path/to/largeInput.json
This solution is not restricted to Ubuntu. It should work on any operating system.
Use the external tool option.
This worked for me
http://www.milosev.com/downloads/websphere/117-linux/ubuntu/454-json-prettifier-for-gedit.html
When attempting to use pandoc to convert JSON based files (.ipynb) from iPython notebook (0.12), I receive an error stating "bad decodeArgs" for the JSON. I suspect that it may be due to the Ubuntu provided version of pandoc that I am using (1.8.1.1). It seems that getting the latest pandoc version requires setting up the Haskell platform which I was not successful doing because of dependency challenges (and really don't want to). I don't want to spend any more time trying to install Haskell if this is not my problem.
Is there a way to get the latest pandoc binaries for Ubuntu without rebuilding it?
Given that iPython notebook is new (and very cool!!), it would be nice to hear about experiences related to translating the JSON to other formats. Perhaps there is a different way to accomplish this other than pandoc.
Regarding your "keeping up to date with Pandoc", I'm afraid you do need Haskell installed. The best way to do this via the Haskell Platform ("HP") package, and then just like with Ruby, it's a lot more consistent to use the environment's package manager for dependencies than your OS. I've had no trouble getting it working, even in Windoze. . .
I'm sure questions to the Haskell mailing list would result in quick help for a platform as mainstream as Debian/Ubuntu, but you might need to manually install a newer version of HP that what's available through the OS package manager.
Once you get HP up and running, the dev Pandoc is dead easy to compile, and git will keep you up to date with the latest - specific instructions here, currently maintained:
https://github.com/jgm/pandoc/wiki/Installing-the-development-version-of-pandoc-1.9
Note that v1.9 has now been officially released if you really don't want to go to the trouble of keeping up to date with the dev cycle, but of course again you won't get it in your OS package manager for quite some time after that (I assume anyway).
==========================
Regarding your attempts to treat JSON as a document syntax:
The best syntax inputs for Pandoc at this point are its native markdown+extensions, and reST (especially for Python people/environments), basically maintained as functionally equivalent, although there may be features available in the former that aren't represented in the latter, since John can just add extensions anytime he wants. AFAIK Pandoc hasn't begun to support the Sphinx extensions (yet?)
The JSON format used internally within Pandoc isn't documented (yet?) but it's the native Haskell data type. As the Thomas K notes, there may be some similarity between how the two tools represent data, but probably not enough to treat either as "just another markup format".
However, if you're working on this, it's easy enough to see what Pandoc looks for in the way of JSON input.
pandoc -t json
compare this to
pandoc -t native
and it's easy to see the specs created by Text.Pandoc.Definition and Text.JSON.Generic
Using Pandoc's internal data representation as input would obviously be more stable than a marked up text stream, and others have expressed a desire for documentation on this and it would be a great contribution to the community.
Please do inform the Pandoc mail list of any work done in this area. The crew there is very responsive, including getting quick feedback from John M (the lead developer) himself directly.
I doubt pandoc or any other tool knows what to do with ipynb files yet (at the time of writing, the IPython notebook was released less than a month ago). JSON is just a generic data structure like XML, not a document format.
We're (IPython) working on tools to export notebooks to other formats, but they're not ready for a proper release yet. If you want to help develop that, see this mailing list thread. Hopefully it will be part of the next IPython release.
I have a binary file to which I'm trying to write however I dont have the file format specification nor have found it using google, I've been looking at the file using a hex editor but so far has only give me a headache, is there a better way to decipher the format of the file so that I can append data to it?
File carving tools such as scalpel won't really help here. They're made for extracting files with known header and/or footer signatures from a memory dump or some larger, composite file.
For your scenario, I would recommend a hex editor with templating capability, like the 010 Editor. This will allow you to name and annotate "fields" in the binary as you learn more about what each part of the file does. Unfortunately, the process of finding out what each field does is mostly manual. As a methodology, just start playing with it. Change some values in your current binary and see what happens. Expect to spend significant time on it, but also enjoy the process!
you may want to search it with a open source forensic application like foremost or scalpel. They will do most of the grunt work for you, you just likely wont learn anything.
I have a binary file. I don't know how it's formatted, I only know it comes from a delphi code.
Does it exist any way to analyze a binary file?
Does it exist any "pattern" to analyze and deserialize the binary content of a file with unknown format?
Try these:
Deserialize data: analyze how it's compiled your exe (try File Analyzer). Try to deserialize the binary data with the language discovered. Then serialize it in a xml format (language-indipendent) that every programming language can understand
Analyze the binary data: try to save various versions of the file with little variation and use a diff program to analyze the meaning of every bit with an hex editor. Use it in conjunction with binary hacking techniques (like How to crack a Binary File Format by Frans Faase)
Reverse Engineer the application: try getting code using reverse engineering tools for the programming language used for build the app (found with File Analyzer). Otherwise use disassembler analysis tool like IDA Pro Disassembler
For my hobby project I had to reverse engineer some old game files. My approaches were:
Have a good hex editor.
Look for readable words in the binary file. Note how their distribution is. If the distance between them is constant you know it is a listing.
Look for 2-3 consequent zeros. Might indicate an int32 value.
Some dwords might be pointers into the file.
Try to identify reoccurring patterns in the file.
Seeing lots of C0-CF might indicate RLE compressed data.
I've developed Hexinator (Window & Linux) and Synalyze It! (macOS) exactly for this purpose. These applications allow you to see the binary files like in other hex editors but additionally you can create a "grammar" with the specifics of a binary file format. The grammar contains all the building blocks and is used to parse the file automatically.
Thus you can keep the knowledge you gain in the analysis and apply it to multiple files simultaneously. You can also color-code the bits and pieces of file formats for a quick overview in the hex editor.
The parsing results are displayed in a tree view where you can also modify the files easily (applying endianness et cetera).
Reverse engineering a binary file when you have some idea of what it represents is a very time consuming process. If you have no idea what it is then it will be even harder.
It is possible though, but you have to have a pretty good reason for doing so.
The first step would be to open it up in a hex editor of your choice and see if you can find any English text to point you in the direction of what the file is even supposed to represent. From there, Google "Reverse Engineering binary files", there are much more knowledgeable people than me that have written guides about it.
The "strings" program from GNU binutils is very useful. It will print the strings of printable characters in a file, quite often giving a clue to what a file contains or a program does.
If the data represents serialized Delphi objects, you should start reading about the Delphi serialization process. If that's the case, I think your best bet would be to load it using Delphi and continue your analysis from the IDE. Some informations about Delphi serialization can be found here.
EDIT: if the file does contain serialized delphi objects, then you should write a small delphi program that loads it, and "convert" the data yourself to something neutral, like xml. If you manage to do this, you should check and see if delphi supports serializing to xml. Then, you could access those objects from any language.
The unix "file" command is really useful - I don't know if there is anything like it in windows. You run it like this:
file myfile.ext
And it spits out a text description based on the magic numbers and data contained therein.
Probably it is contained within cygwin.
If you have access to the application that creates the file, you can apply changes to the application, then save the file and see the effects (Keep in mind that numbers are probably stored in little endian):
First create the file repeatedly. If the files are not binary equal, the current date/time is probably stored in the area where hte differences occur.
Maybe you want to repeat that with the software running under different environments, to see if OS version etc are stored, but this is rather unusual.
Next you can try to change single variables and create several files that only differ in the value of this variable. This helps you identify where this variable is stored.
That way you can also exclude variables that are not stored in the file: If you change them, but the files created are identical, they are not stored.
In order to test the hypotheses you worked out with the steps above, edit one of the files and have the application read it.
If you don't have access to the application itself, I suggest that you forget about it and find another way to solve your problem. There is a very high probability that it will be faster...
If file does not give a meaningful answer, you may want to try TRiD by Marco Pontello to determine whether your data is stored in a known format.
Get the Delphi application and open it in IDA Pro freeware version, and find where it writes the file, and decode how it writes the file that way.
Unless it's plan text.
Do you know the program that uses it? If so you can hook that programs write to file function and get an idea of what data its writing, the size of the data and where.
More Info: http://www.codeproject.com/KB/DLL/Win32APIHooking_Trouble.aspx
Unlike traditional hex editors which only display the raw hex bytes of a file, 010 Editor can also parse a file into a hierarchical structure using a Binary Template. The results of running a Binary Template are much easier to understand and edit than using just the raw hex bytes.
http://www.sweetscape.com/010editor/
Try to open it in a hex editor and analyse.